IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2506.00856.html
   My bibliography  Save this paper

Can AI Master Econometrics? Evidence from Econometrics AI Agent on Expert-Level Tasks

Author

Listed:
  • Qiang Chen
  • Tianyang Han
  • Jin Li
  • Ye Luo
  • Yuxiao Wu
  • Xiaowei Zhang
  • Tuo Zhou

Abstract

Can AI effectively perform complex econometric analysis traditionally requiring human expertise? This paper evaluates AI agents' capability to master econometrics, focusing on empirical analysis performance. We develop an ``Econometrics AI Agent'' built on the open-source MetaGPT framework. This agent exhibits outstanding performance in: (1) planning econometric tasks strategically, (2) generating and executing code, (3) employing error-based reflection for improved robustness, and (4) allowing iterative refinement through multi-round conversations. We construct two datasets from academic coursework materials and published research papers to evaluate performance against real-world challenges. Comparative testing shows our domain-specialized AI agent significantly outperforms both benchmark large language models (LLMs) and general-purpose AI agents. This work establishes a testbed for exploring AI's impact on social science research and enables cost-effective integration of domain expertise, making advanced econometric methods accessible to users with minimal coding skills. Furthermore, our AI agent enhances research reproducibility and offers promising pedagogical applications for econometrics teaching.

Suggested Citation

  • Qiang Chen & Tianyang Han & Jin Li & Ye Luo & Yuxiao Wu & Xiaowei Zhang & Tuo Zhou, 2025. "Can AI Master Econometrics? Evidence from Econometrics AI Agent on Expert-Level Tasks," Papers 2506.00856, arXiv.org, revised Jun 2025.
    • Handle: RePEc:arx:papers:2506.00856
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2506.00856
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Erik Brynjolfsson & Danielle Li & Lindsey Raymond, 2025. "Generative AI at Work," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 140(2), pages 889-942.
    2. Margarita Leib & Nils Köbis & Rainer Michael Rilke & Marloes Hagens & Bernd Irlenbusch, 2023. "Corrupted by Algorithms? How AI-generated and Human-written Advice Shape (Dis)honesty," ECONtribute Discussion Papers Series 251, University of Bonn and University of Cologne, Germany.
    3. Douglas Almond & Kenneth Y. Chay & David S. Lee, 2005. "The Costs of Low Birth Weight," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 120(3), pages 1031-1083.
    4. Gorodnichenko, Yuriy & Pham, Tho & Talavera, Oleksandr, 2025. "Central bank communication on social media: What, to whom, and how?," Journal of Econometrics, Elsevier, vol. 249(PC).
    5. Ahrens, Maximilian & Erdemlioglu, Deniz & McMahon, Michael & Neely, Christopher J. & Yang, Xiye, 2025. "Mind your language: Market responses to central bank speeches," Journal of Econometrics, Elsevier, vol. 249(PC).
    6. Margarita Leib & Nils Kobis & Rainer Michael Rilke & Marloes Hagens & Bernd Irlenbusch, 2023. "Corrupted by Algorithms? How AI-generated and Human-written Advice Shape (Dis)honesty," Papers 2301.01954, arXiv.org.
    7. Card, David & Sullivan, Daniel G, 1988. "Measuring the Effect of Subsidized Training Programs on Movements in and out of Employment," Econometrica, Econometric Society, vol. 56(3), pages 497-530, May.
    8. Leib, Margarita & Köbis, Nils & Rilke, Rainer Michael & Hagens, Marloes & Irlenbusch, Bernd, 2023. "Corrupted by Algorithms? How AI-Generated and Human-Written Advice Shape (Dis)Honesty," IZA Discussion Papers 16293, Institute of Labor Economics (IZA).
    9. Cattaneo, Matias D., 2010. "Efficient semiparametric estimation of multi-valued treatment effects under ignorability," Journal of Econometrics, Elsevier, vol. 155(2), pages 138-154, April.
    10. Anton Korinek, 2023. "Generative AI for Economic Research: Use Cases and Implications for Economists," Journal of Economic Literature, American Economic Association, vol. 61(4), pages 1281-1317, December.
    11. Noailly, Joëlle & Nowzohour, Laura & van den Heuvel, Matthias & Pla, Ireneu, 2024. "Heard the news? Environmental policy and clean investments," Journal of Public Economics, Elsevier, vol. 238(C).
    12. Niu, Yimeng & Wu, Jing & Jiang, Shenyang & Jiang, Zhibin, 2025. "The bullwhip effect in servitized manufacturers," Other publications TiSEM 638f661c-295a-4ada-8c33-a, Tilburg University, School of Economics and Management.
    13. Ali Goli & Amandeep Singh, 2024. "Frontiers: Can Large Language Models Capture Human Preferences?," Marketing Science, INFORMS, vol. 43(4), pages 709-722, July.
    14. Thomas Graeber & Christopher Roth & Florian Zimmermann, 2024. "Stories, Statistics, and Memory," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 139(4), pages 2181-2225.
    15. Peiyao Li & Noah Castelo & Zsolt Katona & Miklos Sarvary, 2024. "Frontiers: Determining the Validity of Large Language Models for Automated Perceptual Analysis," Marketing Science, INFORMS, vol. 43(2), pages 254-266, March.
    16. Anderson, Michael, 2008. "Safety for whom? The effects of light trucks on traffic fatalities," Journal of Health Economics, Elsevier, vol. 27(4), pages 973-989, July.
    17. Yang Chen & Samuel N. Kirshner & Anton Ovchinnikov & Meena Andiappan & Tracy Jenkin, 2025. "A Manager and an AI Walk into a Bar: Does ChatGPT Make Biased Decisions Like We Do?," Manufacturing & Service Operations Management, INFORMS, vol. 27(2), pages 354-368, March.
    18. Ishita Chakraborty & Khai Chiong & Howard Dover & K. Sudhir, 2025. "Can AI and AI-Hybrids Detect Persuasion Skills? Salesforce Hiring with Conversational Video Interviews," Marketing Science, INFORMS, vol. 44(1), pages 30-53, January.
    19. Gmyrek, Pawel, & Lutz, Christoph, & Newlands, Gemma,, 2024. "A Technological Construction of Society comparing GPT-4 and human respondents for occupational evaluation in the UK," ILO Working Papers 995347793502676, International Labour Organization.
    20. Bryan K. Stroube & David M. Waguespack, 2024. "Status and consensus: Heterogeneity in audience evaluations of female‐ versus male‐lead films," Strategic Management Journal, Wiley Blackwell, vol. 45(5), pages 994-1024, May.
    21. Yimeng Niu & Jing Wu & Shenyang Jiang & Zhibin Jiang, 2025. "The Bullwhip Effect in Servitized Manufacturers," Management Science, INFORMS, vol. 71(1), pages 1-20, January.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Buchanan, Joy & Hickman, William, 2024. "Do people trust humans more than ChatGPT?," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 112(C).
    2. Verónica Amarante & Marco Manacorda & Edward Miguel & Andrea Vigorito, 2016. "Do Cash Transfers Improve Birth Outcomes? Evidence from Matched Vital Statistics, Program, and Social Security Data," American Economic Journal: Economic Policy, American Economic Association, vol. 8(2), pages 1-43, May.
    3. Shu Wang & Zijun Yao & Shuhuai Zhang & Jianuo Gai & Tracy Xiao Liu & Songfa Zhong, 2025. "When Experimental Economics Meets Large Language Models: Evidence-based Tactics," Papers 2505.21371, arXiv.org, revised Jul 2025.
    4. Sokbae Lee & Ryo Okui & Yoon†Jae Whang, 2017. "Doubly robust uniform confidence band for the conditional average treatment effect function," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(7), pages 1207-1225, November.
    5. Difang Huang & Jiti Gao & Tatsushi Oka, 2022. "Semiparametric Single-Index Estimation for Average Treatment Effects," Papers 2206.08503, arXiv.org, revised Jan 2025.
    6. Yuan Gao & Dokyun Lee & Gordon Burtch & Sina Fazelpour, 2024. "Take Caution in Using LLMs as Human Surrogates: Scylla Ex Machina," Papers 2410.19599, arXiv.org, revised Jan 2025.
    7. Chen, Le-Yu & Lee, Sokbae, 2023. "Sparse quantile regression," Journal of Econometrics, Elsevier, vol. 235(2), pages 2195-2217.
    8. Samuel Chang & Andrew Kennedy & Aaron Leonard & John A. List, 2024. "12 Best Practices for Leveraging Generative AI in Experimental Research," NBER Working Papers 33025, National Bureau of Economic Research, Inc.
    9. Dhruv Grewal & Cinthia B. Satornino & Thomas Davenport & Abhijit Guha, 2025. "How generative AI Is shaping the future of marketing," Journal of the Academy of Marketing Science, Springer, vol. 53(3), pages 702-722, May.
    10. Jianxuan Liu & Yanyuan Ma & Lan Wang, 2018. "An alternative robust estimator of average treatment effect in causal inference," Biometrics, The International Biometric Society, vol. 74(3), pages 910-923, September.
    11. Kiwoong Yoo & Michael Haenlein & Kelly Hewett, 2025. "A whole new world, a new fantastic point of view: Charting unexplored territories in consumer research with generative artificial intelligence," Journal of the Academy of Marketing Science, Springer, vol. 53(3), pages 723-759, May.
    12. Xu, Wenfu & Tan, Zhiqiang, 2024. "High-dimensional model-assisted inference for treatment effects with multi-valued treatments," Journal of Econometrics, Elsevier, vol. 244(1).
    13. Angelico, Cristina, 2024. "The green transition and firms' expectations on future prices: Survey evidence," Journal of Economic Behavior & Organization, Elsevier, vol. 221(C), pages 519-543.
    14. Rothe, Christoph & Firpo, Sergio, 2013. "Semiparametric Estimation and Inference Using Doubly Robust Moment Conditions," IZA Discussion Papers 7564, Institute of Labor Economics (IZA).
    15. Du, Ke & Jia, Fu & Chen, Lujie, 2025. "The impact of customers’ MD&A sentiment on the bullwhip effect: A dyadic buyer-supplier perspective," International Journal of Production Economics, Elsevier, vol. 284(C).
    16. Paola Cillo & Gaia Rubera, 2025. "Generative AI in innovation and marketing processes: A roadmap of research opportunities," Journal of the Academy of Marketing Science, Springer, vol. 53(3), pages 684-701, May.
    17. Jürgensmeier, Lukas & Skiera, Bernd, 2024. "Generative AI for scalable feedback to multimodal exercises," International Journal of Research in Marketing, Elsevier, vol. 41(3), pages 468-488.
    18. Sergio Firpo & Cristine Pinto, 2016. "Identification and Estimation of Distributional Impacts of Interventions Using Changes in Inequality Measures," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 31(3), pages 457-486, April.
    19. Difang Huang & Jiti Gao & Tatsushi Oka, 2025. "Semiparametric single-index estimation for average treatment effects," Econometric Reviews, Taylor & Francis Journals, vol. 44(6), pages 843-885, July.
    20. Riccardo Di Francesco, 2024. "Aggregation Trees," Papers 2410.11408, arXiv.org.

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2506.00856. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.