IDEAS home Printed from https://ideas.repec.org/p/jgu/wpaper/2414.html
   My bibliography  Save this paper

A Comparison of Large Language Models and Genetic Programming for Program Synthesis

Author

Listed:
  • Dominik Sobania

    (Johannes-Gutenberg University, Germany)

  • Justyna Petke

    (University College London, United Kingdom)

  • Martin Briesch

    (Johannes-Gutenberg University, Germany)

  • Franz Rothlauf

    (Johannes-Gutenberg University, Germany)

Abstract

Large language models have recently become known for their ability to generate computer programs, especially through tools such as GitHub Copilot, a domain where genetic programming has been very successful so far. Although they require different inputs (free-text vs. input/output examples) their goal is the same – program synthesis. Therefore, in this work we compare how well GitHub Copilot and genetic programming perform on common program synthesis benchmark problems. We study the structure and diversity of the generated programs by using well-known software metrics. We find that GitHub Copilot and genetic programming solve a similar number of benchmark problems (85.2% vs. 77.8%, respectively). We find that GitHub Copilot generated smaller and less complex programs as genetic programming, while genetic programming is able to find new and unique problem solving strategies. This increase in diversity of solutions comes at a cost. When analyzing the success rates for 100 runs per problem, GitHub Copilot outperforms genetic programming on over 50% of the problems.

Suggested Citation

  • Dominik Sobania & Justyna Petke & Martin Briesch & Franz Rothlauf, 2024. "A Comparison of Large Language Models and Genetic Programming for Program Synthesis," Working Papers 2414, Gutenberg School of Management and Economics, Johannes Gutenberg-Universität Mainz.
    • Handle: RePEc:jgu:wpaper:2414
    as

    Download full text from publisher

    File URL: https://download.uni-mainz.de/RePEc/pdf/Discussion_Paper_2414.pdf
    File Function: First version, 2024
    Download Restriction: no
    ---><---

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:jgu:wpaper:2414. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Research Unit IPP (email available below). General contact details of provider: https://edirc.repec.org/data/vlmaide.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.