IDEAS home Printed from https://ideas.repec.org/p/arx/papers/2604.03338.html

The Ideation Bottleneck: Decomposing the Quality Gap Between AI-Generated and Human Economics Research

Author

Listed:
  • Ning Li

Abstract

Autonomous AI systems can now generate complete economics research papers, but they substantially underperform human-authored publications in head-to-head comparisons. This paper decomposes the quality gap into two independent components: research idea quality and execution quality. Using a two-model ensemble of fine-tuned language models trained on publication decisions (Gong, Li, and Zhou, 2026) to evaluate idea quality and a comprehensive six-dimension rubric assessed by Gemini 3.1 Flash Lite -- the same model family used as the APE tournament judge, ensuring methodological consistency -- to evaluate execution quality, we analyze 953 economics papers -- 912 AI-generated papers from the APE project and 41 human papers published in the American Economic Review and AEJ: Economic Policy. The idea quality gap is large (Cohen's d = 2.23, p

Suggested Citation

  • Ning Li, 2026. "The Ideation Bottleneck: Decomposing the Quality Gap Between AI-Generated and Human Economics Research," Papers 2604.03338, arXiv.org.
    • Handle: RePEc:arx:papers:2604.03338
    as

    Download full text from publisher

    File URL: http://arxiv.org/pdf/2604.03338
    File Function: Latest version
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Goodman-Bacon, Andrew, 2021. "Difference-in-differences with variation in treatment timing," Journal of Econometrics, Elsevier, vol. 225(2), pages 254-277.
    2. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    3. Joshua D. Angrist & Jörn-Steffen Pischke, 2009. "Mostly Harmless Econometrics: An Empiricist's Companion," Economics Books, Princeton University Press, edition 1, number 8769, December.
    4. Ziqin Gong & Ning Li & Huaikang Zhou, 2026. "LLMs learn scientific taste from institutional traces across the social sciences," Papers 2603.16659, arXiv.org, revised May 2026.
    5. Anton Korinek, 2023. "Generative AI for Economic Research: Use Cases and Implications for Economists," Journal of Economic Literature, American Economic Association, vol. 61(4), pages 1281-1317, December.
    6. Jonathan Roth, 2022. "Pretest with Caution: Event-Study Estimates after Testing for Parallel Trends," American Economic Review: Insights, American Economic Association, vol. 4(3), pages 305-322, September.
    7. Emily Oster, 2019. "Unobservable Selection and Coefficient Stability: Theory and Evidence," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 37(2), pages 187-204, April.
    8. Callaway, Brantly & Sant’Anna, Pedro H.C., 2021. "Difference-in-Differences with multiple time periods," Journal of Econometrics, Elsevier, vol. 225(2), pages 200-230.
    9. Robert Novy-Marx & Mihail Z. Velikov, 2025. "AI-Powered (Finance) Scholarship," NBER Working Papers 33363, National Bureau of Economic Research, Inc.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Arne Henningsen & Guy Low & David Wuepper & Tobias Dalhaus & Hugo Storm & Dagim Belay & Stefan Hirsch, 2024. "Estimating Causal Effects with Observational Data: Guidelines for Agricultural and Applied Economists," IFRO Working Paper 2024/03, University of Copenhagen, Department of Food and Resource Economics.
    2. Mark Kattenberg & Bas Scheer & Jurre Thiel, 2023. "Causal forests with fixed effects for treatment effect heterogeneity in difference-in-differences," CPB Discussion Paper 452, CPB Netherlands Bureau for Economic Policy Analysis.
    3. Yang, Xiaoyin & Zhang, Xinmin & Zang, Hong & Chu, Hongcong & Feng, Chao, 2025. "From shadows to spotlight: How big data bureaus unveil corporate R&D gaming?," Finance Research Letters, Elsevier, vol. 86(PG).
    4. Roth, Jonathan & Sant’Anna, Pedro H.C. & Bilinski, Alyssa & Poe, John, 2023. "What’s trending in difference-in-differences? A synthesis of the recent econometrics literature," Journal of Econometrics, Elsevier, vol. 235(2), pages 2218-2244.
    5. Howell, Anthony, 2025. "Corrigendum to “Rural road stimulus and the role of matching mandates on economic recovery in China” [J. Dev. Econ. 166, (January 2024), 103211]," Journal of Development Economics, Elsevier, vol. 172(C).
    6. Cao, June & Huang, Zijie & Kristanto, Ari Budi, 2025. "From bytes to blooms: Tech-driven transformation and green revenues," Energy Economics, Elsevier, vol. 144(C).
    7. Stefan Bauernschuster & Michael Grimm & Cathy M. Hajo, 2023. "The Impact of Margaret Sanger’s Birth Control Clinics on Early 20th Century U.S. Fertility and Mortality," CESifo Working Paper Series 10421, CESifo.
    8. Bradford, Ashley C. & Fu, Wei & You, Shijun, 2024. "The devastating dance between opioid and housing crises: Evidence from OxyContin reformulation," Journal of Health Economics, Elsevier, vol. 98(C).
    9. Wang, Lisha & Pan, Minjie & Qian, Xinlei & Lv, Kangjuan, 2025. "Do specialized courts matter? Environmental judiciary and corporate emissions in China," Energy Policy, Elsevier, vol. 199(C).
    10. Maclean, J. Catherine & Pichler, Stefan & Ziebarth, Nicolas R., 2020. "Mandated Sick Pay: Coverage, Utilization, and Welfare Effects," IZA Discussion Papers 13132, IZA Network @ LISER.
    11. André A. Castro & Philipp Ehrl, 2025. "Lean on me, firm: evidence from a management consulting program," Small Business Economics, Springer, vol. 65(1), pages 61-78, June.
    12. Siverskog, Jonathan & Henriksson, Martin, 2022. "The health cost of reducing hospital bed capacity," Social Science & Medicine, Elsevier, vol. 313(C).
    13. Ashesh Rambachan & Jonathan Roth, 2020. "Design-Based Uncertainty for Quasi-Experiments," Papers 2008.00602, arXiv.org, revised Jun 2025.
    14. Chy, Mahfuz & Kyung, Hoyoun, 2023. "The effect of bond market transparency on bank loan contracting," Journal of Accounting and Economics, Elsevier, vol. 75(2).
    15. Li, Tongxia & Ang, Tze Chuan ‘Chewie’ & Lu, Chun, 2023. "Employment protection and the provision of trade credit," Journal of Banking & Finance, Elsevier, vol. 155(C).
    16. Li, Daiyue & Jin, Yanhong & Cheng, Mingwang, 2024. "Unleashing the power of industrial robotics on firm productivity: Evidence from China," Journal of Economic Behavior & Organization, Elsevier, vol. 224(C), pages 500-520.
    17. Yan, Wenying & Chen, Yusheng & Wang, Yanmei, 2025. "Efficiency improvement effect of clean energy transformation —A quasi-natural experiment based on China's clean heating policy," Energy, Elsevier, vol. 334(C).
    18. Prem, Mounu & Purroy, Miguel E. & Vargas, Juan F., 2025. "Landmines: The local effects of demining," Journal of Public Economics, Elsevier, vol. 247(C).
    19. Philipp Barteska & Jay Euijung Lee, 2024. "Bureaucrats and the Korean export miracle," Discussion Papers 2024-11, Nottingham Interdisciplinary Centre for Economic and Political Research (NICEP).
    20. repec:osf:socarx:v75aw_v1 is not listed on IDEAS
    21. Goeyvaerts, Geert, 2023. "Reconstructing cities: Stimulating redevelopment through the tax code," Regional Science and Urban Economics, Elsevier, vol. 99(C).

    More about this item

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:arx:papers:2604.03338. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: arXiv administrators (email available below). General contact details of provider: http://arxiv.org/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.