IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/35782.html

An LLM Workflow That Reproduces, Improves, and Extends Published Economics Research

Author

Listed:
  • Matthew Schwartz
  • Isaiah Andrews
  • Jesse M. Shapiro

Abstract

We introduce an open-source workflow that enables an LLM to reproduce, improve, and extend an economics article using the article’s published replication package. First, the workflow attempts to reproduce the original calculations, checks for discrepancies with published findings, and performs automated sensitivity analysis. Across 4,452 published replication packages for five economics journals, the workflow flags discrepancies in 3,460 articles or their appendices. Second, the workflow improves the original calculations by using a different implementation or algorithm. In 496 articles, the workflow is able to reduce a calculation’s computation time, at similar or greater accuracy, by more than a factor of 10. Third, the workflow extends the original analysis. In 923 articles, the workflow develops an extension that does not appear in the original article and that is aligned with the original article’s goals and assumptions.

Suggested Citation

  • Matthew Schwartz & Isaiah Andrews & Jesse M. Shapiro, 2026. "An LLM Workflow That Reproduces, Improves, and Extends Published Economics Research," NBER Working Papers 35782, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:35782
    Note: TWP
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w35782.pdf
    Download Restriction: Access to the full text is generally limited to series subscribers, however if the top level domain of the client browser is in a developing country or transition economy free access is provided. More information about subscriptions and free access is available at http://www.nber.org/wwphelp.html. Free access is also available to older working papers.
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Miloš Fišar & Ben Greiner & Christoph Huber & Elena Katok & Ali I. Ozkes, 2024. "Reproducibility in Management Science," Management Science, INFORMS, vol. 70(3), pages 1343-1356, March.
    2. Leo Kaas & Etienne Lalé & Nawid Siassi, 2026. "Job Ladder and Wealth Dynamics in General Equilibrium," Econometrica, Econometric Society, vol. 94(4), pages 1449-1485, July.
    3. Christophe Pérignon & Olivier Akmansoy & Christophe Hurlin & Anna Dreber & Felix Holzmeister & Jürgen Huber & Magnus Johannesson & Michael Kirchler & Albert J Menkveld & Michael Razen & Utz Weitzel, 2024. "Computational Reproducibility in Finance: Evidence from 1,000 Tests," The Review of Financial Studies, Society for Financial Studies, vol. 37(11), pages 3558-3593.
    4. Daniil A. Boiko & Robert MacKnight & Ben Kline & Gabe Gomes, 2023. "Autonomous chemical research with large language models," Nature, Nature, vol. 624(7992), pages 570-578, December.
    5. Trieu H. Trinh & Yuhuai Wu & Quoc V. Le & He He & Thang Luong, 2024. "Author Correction: Solving olympiad geometry without human demonstrations," Nature, Nature, vol. 627(8004), pages 8-8, March.
    6. Mariana Carrera & Heather Royer & Mark Stehr & Justin Sydnor & Dmitry Taubinsky, 2022. "Who Chooses Commitment? Evidence and Welfare Implications," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 89(3), pages 1205-1244.
    7. Trieu H. Trinh & Yuhuai Wu & Quoc V. Le & He He & Thang Luong, 2024. "Solving olympiad geometry without human demonstrations," Nature, Nature, vol. 625(7995), pages 476-482, January.
    8. B.D. McCullough & Kerry Anne McGeary & Teresa D. Harrison, 2008. "Do economics journal archives promote replicable research?," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 41(4), pages 1406-1420, November.
    9. Richard Anderson & William Greene & B. D. McCullough & H. D. Vinod, 2008. "The role of data/code archives in the future of economic research," Journal of Economic Methodology, Taylor & Francis Journals, vol. 15(1), pages 99-119.
    10. Ufuk Akcigit & Jeremy Pearce & Marta Prato, 2025. "Tapping into Talent: Coupling Education and Innovation Policies for Economic Growth," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 92(2), pages 696-736.
    11. Andrew C. Chang & Phillip Li, 2022. "Is Economics Research Replicable? Sixty Published Papers From Thirteen Journals Say “Often Notâ€," Critical Finance Review, now publishers, vol. 11(1), pages 185-206, February.
    12. Abel Brodeur & Derek Mikola & Nikolai Cook & Lenka Fiala & Thomas Brailey & Ryan Briggs & Alexandra de Gendre & Yannick Dupraz & Jacopo Gabani & Romain Gauriot & Joanne Haddad & Goncalo Lima & Jörg An, 2026. "Publisher Correction: Reproducibility and robustness of economics and political science research," Nature, Nature, vol. 653(8114), pages 7-7, May.
    13. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Hai Lu & Yichen Luo & Luping Yu, 2026. "How Well Do LLMs Perform in Peer Review? Evidence from Finance and Accounting," Papers 2610.18435, arXiv.org.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ferman, Bruno & Finamor, Lucas, 2025. "There must be an error here! Experimental evidence on coding errors' biases," I4R Discussion Paper Series 266, The Institute for Replication (I4R).
    2. Balafoutas, Loukas & Celse, Jeremy & Karakostas, Alexandros & Umashev, Nicholas, 2025. "Incentives and the replication crisis in social sciences: A critical review of open science practices," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 114(C).
    3. Christophe Hurlin & Jean-Edouard Colliard & Christophe Pérignon, 2024. "The Economics of Computational Reproducibility," Working Papers hal-04759240, HAL.
    4. Tom L. Dudda & Lars Hornuf, 2025. "The Perks and Perils of Machine Learning in Business and Economic Research," CESifo Working Paper Series 11721, CESifo.
    5. Anna Dreber & Magnus Johannesson, 2025. "A framework for evaluating reproducibility and replicability in economics," Economic Inquiry, Western Economic Association International, vol. 63(2), pages 338-356, April.
    6. Guillaume Coqueret & Joan Llull & Florian Oswald & Christophe Pérignon & Christoph Scheuch & Lars Vilhuber, 2026. "Randomness In Large Language Models: What Researchers Need to Know (And Report)," Working Papers hal-05730402, HAL.
    7. Mueller-Langer, Frank & Andreoli-Versbach, Patrick, 2018. "Open access to research data: Strategic delay and the ambiguous welfare effects of mandatory data disclosure," Information Economics and Policy, Elsevier, vol. 42(C), pages 20-34.
    8. Maurizio Canavari & Andreas C. Drichoutis & Jayson L. Lusk & Rodolfo M. Nayga, Jr., 2018. "How to run an experimental auction: A review of recent advances," Working Papers 2018-5, Agricultural University of Athens, Department Of Agricultural Economics.
    9. Wenhao Zhou & Faqiang Liu & Hao Zheng & Rong Zhao, 2025. "Mitigating data bias and ensuring reliable evaluation of AI models with shortcut hull learning," Nature Communications, Nature, vol. 16(1), pages 1-15, December.
    10. Jörg Ankel‐Peters & Nathan Fiala & Florian Neubauer, 2025. "Is economics self‐correcting? Replications in the American Economic Review," Economic Inquiry, Western Economic Association International, vol. 63(2), pages 463-485, April.
    11. Le Zhang & Andreas Ortmann, 2012. "A reproduction and replication of Engel’s meta-study of dictator game experiments," Discussion Papers 2012-44, School of Economics, The University of New South Wales.
    12. Lukas Fink & Jan Marcus, 2025. "Replication code availability over time and across fields: Evidence from the German Socio‐Economic Panel," Economic Inquiry, Western Economic Association International, vol. 63(2), pages 357-386, April.
    13. repec:bla:germec:v:11:y:2010:i::p:404-417 is not listed on IDEAS
    14. Miloš Fišar & Ben Greiner & Christoph Huber & Elena Katok & Ali I. Ozkes, 2024. "Reproducibility in Management Science," Management Science, INFORMS, vol. 70(3), pages 1343-1356, March.
    15. Cristina Blanco-Perez & Abel Brodeur, 2020. "Publication Bias and Editorial Statement on Negative Findings," The Economic Journal, Royal Economic Society, vol. 130(629), pages 1226-1247.
    16. Zhou, Qiang & Yang, Yefei & Ma, Fangfang & Cheng, T.C. Edwin, 2026. "Multi-agent deep reinforcement learning for ordering and inventory allocation in a decentralized two-echelon dual-channel supply chain," International Journal of Production Economics, Elsevier, vol. 299(C).
    17. Michael Klesel & H. Felix Wittmann, 2025. "Retrieval-Augmented Generation (RAG)," Business & Information Systems Engineering: The International Journal of WIRTSCHAFTSINFORMATIK, Springer;Gesellschaft für Informatik e.V. (GI), vol. 67(4), pages 551-561, August.
    18. Michael A. Clemens, 2017. "The Meaning Of Failed Replications: A Review And Proposal," Journal of Economic Surveys, Wiley Blackwell, vol. 31(1), pages 326-342, February.
    19. Vlaeminck, Sven, 2013. "Data Management in Scholarly Journals and Possible Roles for Libraries - Some Insights from EDaWaX," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 23(1), pages 49-79.
    20. Hasso, Tim & Brosnan, Mark & Ali, Searat & Chai, Daniel, 2026. "Perceived problems, causes, and solutions of finance research reproducibility and replicability: A pre-registered study," Pacific-Basin Finance Journal, Elsevier, vol. 98(C).
    21. Brodeur, Abel & Baki, Ghina Abdul & Fazio, Luna, 2026. "Mass Reproducibility in Psychological Science," I4R Discussion Paper Series 326, The Institute for Replication (I4R).

    More about this item

    JEL classification:

    • C80 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:35782. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.