IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2603.17381.html

An Auditable AI Agent Loop for Empirical Economics: A Case Study in Forecast Combination

Author

Listed:
  • Minchul Shin

Abstract

AI coding agents make empirical specification search fast and cheap, but they also widen hidden researcher degrees of freedom. Building on an open-source agent-loop architecture, this paper adapts that framework to an empirical economics workflow and adds a post-search holdout evaluation. In a forecast-combination illustration, multiple independent agent runs outperform standard benchmarks in the original rolling evaluation, but not all continue to do so on a post-search holdout. Logged search and holdout evaluation together make adaptive specification search more transparent and help distinguish robust improvements from sample-specific discoveries.

Suggested Citation

  • Minchul Shin, 2026. "An Auditable AI Agent Loop for Empirical Economics: A Case Study in Forecast Combination," Papers 2603.17381, arXiv.org, revised Mar 2026.
    • Handle: RePEc:arx:papers:2603.17381
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2603.17381
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Diebold, Francis X. & Shin, Minchul, 2019. "Machine learning for regularized survey forecast combination: Partially-egalitarian LASSO and its derivatives," International Journal of Forecasting, Elsevier, vol. 35(4), pages 1679-1691.
    2. Edward Miguel, 2021. "Evidence on Research Transparency in Economics," Journal of Economic Perspectives, American Economic Association, vol. 35(3), pages 193-214, Summer.
    3. Herbert Dawid & Philipp Harting & Hankui Wang & Zhongli Wang & Jiachen Yi, 2025. "Agentic Workflows for Economic Research: Design and Implementation," Papers 2504.09736, arXiv.org.
    4. Leamer, Edward E, 1983. "Let's Take the Con Out of Econometrics," American Economic Review, American Economic Association, vol. 73(1), pages 31-43, March.
    5. Minchul Shin & Nathan Schor, 2026. "ForeComp: An R Package for Comparing Predictive Accuracy Using Fixed-Smoothing Asymptotics," Papers 2603.07458, arXiv.org.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Cho, Seo-young & Vadlamannati, Krishna Chaitanya, 2010. "Compliance for big brothers: An empirical analysis on the impact of the anti-trafficking protocol," University of Göttingen Working Papers in Economics 118, University of Goettingen, Department of Economics.
    2. Ho, Ron Yiu-wah & Strange, Roger & Piesse, Jenifer, 2006. "On the conditional pricing effects of beta, size, and book-to-market equity in the Hong Kong market," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 16(3), pages 199-214, July.
    3. Jing Xing, 2011. "Does tax structure affect economic growth? Empirical evidence from OECD countries," Working Papers 1120, Oxford University Centre for Business Taxation.
    4. David Card, 2022. "Design-Based Research in Empirical Microeconomics," American Economic Review, American Economic Association, vol. 112(6), pages 1773-1781, June.
    5. Marc-Olivier Bessette & Mariame Dioubate & Myriane Hébert & Miriam Elsie Kuimi Tchana & Laura Morissette & Jean-Charles Toupin & Raoul Yaro & Maurice Doyon, 2020. "La présence de biais cognitifs en analyse économique : une étude de cas," CIRANO Working Papers 2020s-12, CIRANO.
    6. Diebold, Francis X. & Rudebusch, Glenn D., 2022. "Probability assessments of an ice-free Arctic: Comparing statistical and climate model projections," Journal of Econometrics, Elsevier, vol. 231(2), pages 520-534.
    7. Terrence Hallahan & Robert Faff, 2001. "Induced persistence or reversals in fund performance?: the effect of survivorship bias," Applied Financial Economics, Taylor & Francis Journals, vol. 11(2), pages 119-126.
    8. Christoph Engel, 2006. "The Difficult Reception of Rigorous Descriptive Social Science in the Law," Discussion Paper Series of the Max Planck Institute for Behavioral Economics 2006_1, Max Planck Institute for Behavioral Economics.
    9. Uniejewski, Bartosz & Maciejowska, Katarzyna, 2023. "LASSO principal component averaging: A fully automated approach for point forecast pooling," International Journal of Forecasting, Elsevier, vol. 39(4), pages 1839-1852.
    10. Durlauf, Steven N., 2001. "Manifesto for a growth econometrics," Journal of Econometrics, Elsevier, vol. 100(1), pages 65-69, January.
    11. Christopher J. Ruhm, 2019. "Shackling the Identification Police?," Southern Economic Journal, John Wiley & Sons, vol. 85(4), pages 1016-1026, April.
    12. Philippe Goulet Coulombe & Maxime Leroux & Dalibor Stevanovic & Stéphane Surprenant, 2022. "How is machine learning useful for macroeconomic forecasting?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 37(5), pages 920-964, August.
    13. Asad Zaman, 2012. "Methodological Mistakes and Econometric Consequences," International Econometric Review (IER), Economic Research Association, vol. 4(2), pages 99-122, September.
    14. R Burger & S du Plessis, 2011. "Examining the Robustness of Competing Explanations of Slow Growth in African Countries," Studies in Economics and Econometrics, Taylor & Francis Journals, vol. 35(3), pages 21-47, December.
    15. Wen, Danyan & Liu, Li & Wang, Yudong & Zhang, Yaojie, 2022. "Forecasting crude oil market returns: Enhanced moving average technical indicators," Resources Policy, Elsevier, vol. 76(C).
    16. Stanley, T. D. & Doucouliagos, Chris, 2019. "Practical Significance, Meta-Analysis and the Credibility of Economics," IZA Discussion Papers 12458, IZA Network @ LISER.
    17. Cho, Seo-Young & Vadlamannati, Krishna Chaitanya, 2012. "Compliance with the Anti-trafficking Protocol," European Journal of Political Economy, Elsevier, vol. 28(2), pages 249-265.
    18. Cohen, Wesley M. & Fjeld, Jon, 2016. "The three legs of a stool: Comment on Richard Nelson, “The sciences are different and the differences matter”," Research Policy, Elsevier, vol. 45(9), pages 1708-1712.
    19. Paldam, Martin, 2015. "Meta-analysis in a nutshell: Techniques and general findings," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy, vol. 9, pages 1-14.
    20. Carmen Fernandez & Eduardo Ley & Mark F. J. Steel, 2001. "Model uncertainty in cross-country growth regressions," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 16(5), pages 563-576.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2603.17381. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.