IDEAS home Printed from https://ideas.repec.org/p/cte/werepe/45949.html
   My bibliography  Save this paper

Assessing ChatGPT's ability to detect and correct programming errors in stata do-files

Author

Listed:
  • Mora, Ricardo

Abstract

This paper evaluates the efectiveness of ChatGPT in debugging Stata, a proprietary econometric software, with a focus on accuracy, hallucination tendencies, and the presence of illusory expertise. By systematically testing ChatGPT-3.5-turbo and ChatGPT-4o across diferent error types and user expertise levels, the study finds that while ChatGPT signifcantly improves debugging performance in open-book mode, its accuracy remainshighly dependent on error complexity. Hallucinations, where the model generates plausible but incorrect error explanations, are particularly frequent in syntax errors and persist even with access to error logs. Advanced scripts present the greatest challenge, with ChatGPT- 4o achieving only a 35.7 percent success rate and exhibiting a 25 percent hallucination rate despite access to error messages. The findings align with broader research on largelanguage models in coding assistance, demonstrating that ChatGPT struggles with complex debugging tasks in proprietary software environments like Stata, where integration is restricted.

Suggested Citation

  • Mora, Ricardo, 2025. "Assessing ChatGPT's ability to detect and correct programming errors in stata do-files," UC3M Working papers. Economics 45949, Universidad Carlos III de Madrid. Departamento de Economía.
    • Handle: RePEc:cte:werepe:45949
    as

    Download full text from publisher

    File URL: https://e-archivo.uc3m.es/rest/api/core/bitstreams/5cba974a-cebc-488f-9eb1-91c3f0839c68/content
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Andrea Bonaccorsi & Silvia Giannangeli & Cristina Rossi, 2006. "Entry Strategies Under Competing Standards: Hybrid Business Models in the Open Source Software Industry," Management Science, INFORMS, vol. 52(7), pages 1085-1098, July.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Bitzer, Jürgen & Geishecker, Ingo, 2010. "Who contributes voluntarily to OSS? An investigation among German IT employees," Research Policy, Elsevier, vol. 39(1), pages 165-172, February.
    2. Fabio M. Manenti & Stefano Comino & Marialaura Parisi, 2005. "From Planning to Mature: on the Determinants of Open Source Take-Off," Industrial Organization 0507006, University Library of Munich, Germany, revised 29 Sep 2005.
    3. Emmanuelle Reuter, 2022. "Hybrid business models in the sharing economy: The role of business model design for managing the environmental paradox," Business Strategy and the Environment, Wiley Blackwell, vol. 31(2), pages 603-618, February.
    4. Stam, Wouter, 2009. "When does community participation enhance the performance of open source software companies?," Research Policy, Elsevier, vol. 38(8), pages 1288-1299, October.
    5. Gauguier, Jean-Jacques, 2009. "L’industrialisation de l’Open Source," Economics Thesis from University Paris Dauphine, Paris Dauphine University, number 123456789/4388 edited by Toledano, Joëlle.
    6. Michael Fritsch & Sebastian von Engelhardt, 2010. "Who Starts with Open Source? Institutional Choice of Start-Ups in the German ICT Sector," Jena Economics Research Papers 2010-049, Friedrich-Schiller-University Jena.
    7. Comino, Stefano & Manenti, Fabio M. & Parisi, Maria Laura, 2007. "From planning to mature: On the success of open source projects," Research Policy, Elsevier, vol. 36(10), pages 1575-1586, December.
    8. Hess, Thomas (Ed.), 2012. "Geschäftsmodelle als Thema der Wirtschaftsinformatik," Working Papers 1/2012, University of Munich, Munich School of Management, Institute for Information Systems and New Media.
    9. Andrea Fosfuri & Marco S. Giarratana & Alessandra Luzzi, 2008. "The Penguin Has Entered the Building: The Commercialization of Open Source Software Products," Organization Science, INFORMS, vol. 19(2), pages 292-305, April.
    10. Amel Charleux & Anne Mione, 2018. "Open source editing business models; the case of software [Les business models de l’édition open source ; Le cas des logiciels]," Post-Print hal-01988127, HAL.
    11. A. Yalta & A. Yalta, 2010. "Should Economists Use Open Source Software for Doing Research?," Computational Economics, Springer;Society for Computational Economics, vol. 35(4), pages 371-394, April.
    12. Cerquera Dussán, Daniel & Müller, Bettina, 2009. "Open Source, ICT infrastructure and firm performance," ZEW Discussion Papers 09-089, ZEW - Leibniz Centre for European Economic Research.
    13. Caulkins, Jonathan P. & Feichtinger, Gustav & Grass, Dieter & Hartl, Richard F. & Kort, Peter M. & Seidl, Andrea, 2013. "When to make proprietary software open source," Journal of Economic Dynamics and Control, Elsevier, vol. 37(6), pages 1182-1194.
    14. Pascale Trompette & Valérie Chanal & Cédric Pelissier, 2008. "Crowdsourcing as a way to access external knowledge for innovation," Post-Print halshs-00367373, HAL.
    15. Sheen S. Levine & Michael J. Prietula, 2014. "Open Collaboration for Innovation: Principles and Performance," Organization Science, INFORMS, vol. 25(5), pages 1414-1433, October.
    16. Marta Widz & Nadine Kammerlander, 2023. "Entrepreneurial exit intentions in emerging economies: a neoinstitutional perspective," Small Business Economics, Springer, vol. 60(2), pages 615-638, February.
    17. Kuk, George & Schaarschmidt, Mario & Homscheid, Dirk, 2024. "All of the same breed? A networking perspective of private-collective innovation," Journal of Business Research, Elsevier, vol. 172(C).
    18. Massimo G. Colombo & Douglas Cumming & Ali Mohammadi & Cristina Rossi-Lamastra & Anu Wadhwa, 2016. "Open business models and venture capital finance," Industrial and Corporate Change, Oxford University Press and the Associazione ICC, vol. 25(2), pages 353-370.
    19. Nicolai j. Foss & Lars Frederiksen & Francesco Rullani, 2016. "Problem‐formulation and problem‐solving in self‐organized communities: How modes of communication shape project behaviors in the free open‐source software community," Strategic Management Journal, Wiley Blackwell, vol. 37(13), pages 2589-2610, December.
    20. Colombo, Massimo G. & Piva, Evila & Rossi-Lamastra, Cristina, 2014. "Open innovation and within-industry diversification in small and medium enterprises: The case of open source software firms," Research Policy, Elsevier, vol. 43(5), pages 891-902.

    More about this item

    Keywords

    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cte:werepe:45949. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Ana Poveda (email available below). General contact details of provider: http://www.eco.uc3m.es/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.