IDEAS home Printed from https://ideas.repec.org/a/bla/ecinqu/v63y2025i2p463-485.html

Is economics self‐correcting? Replications in the American Economic Review

Author

Listed:
  • Jörg Ankel‐Peters
  • Nathan Fiala
  • Florian Neubauer

Abstract

This paper reviews the impact of replications published as comments in the American Economic Review between 2010 and 2020. We examine their citations and influence on the original papers' (OPs) subsequent citations. Our results show that comments are barely cited, and they do not affect the OP's citations—even if the comment diagnoses substantive problems. Furthermore, we conduct an opinion survey among replicators and authors and find that there often is no consensus on whether the OP's contribution sustains. We conclude that the economics literature does not self‐correct, and that robustness and replicability are hard to define in economics.

Suggested Citation

  • Jörg Ankel‐Peters & Nathan Fiala & Florian Neubauer, 2025. "Is economics self‐correcting? Replications in the American Economic Review," Economic Inquiry, Western Economic Association International, vol. 63(2), pages 463-485, April.
    • Handle: RePEc:bla:ecinqu:v:63:y:2025:i:2:p:463-485
    DOI: 10.1111/ecin.13222
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/ecin.13222
    Download Restriction: no
    File URL: https://libkey.io/10.1111/ecin.13222?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    2. Dorothe Bonjour & Lynn F. Cherkas & Jonathan E. Haskel & Denise D. Hawkes & Tim D. Spector, 2003. "Returns to Education: Evidence from U.K. Twins," American Economic Review, American Economic Association, vol. 93(5), pages 1799-1812, December.
    3. Raj Chetty & John N. Friedman & Jonah E. Rockoff, 2014. "Measuring the Impacts of Teachers I: Evaluating Bias in Teacher Value-Added Estimates," American Economic Review, American Economic Association, vol. 104(9), pages 2593-2632, September.
    4. John P. A. Ioannidis & T. D. Stanley & Hristos Doucouliagos, 2017. "The Power of Bias in Economics Research," Economic Journal, Royal Economic Society, vol. 127(605), pages 236-265, October.
    5. repec:isu:genstf:201710010700001659 is not listed on IDEAS
    6. Christoph Brunner & Colin F. Camerer & Jacob K. Goeree, 2011. "Stationary Concepts for Experimental 2 X 2 Games: Comment," American Economic Review, American Economic Association, vol. 101(2), pages 1029-1040, April.
    7. Teplitskiy, Misha & Duede, Eamon & Menietti, Michael & Lakhani, Karim R., 2022. "How status of research papers affects the way they are read and cited," Research Policy, Elsevier, vol. 51(4).
    8. Edward Miguel, 2021. "Evidence on Research Transparency in Economics," Journal of Economic Perspectives, American Economic Association, vol. 35(3), pages 193-214, Summer.
    9. Daron Acemoglu & Simon Johnson & James A. Robinson, 2001. "The Colonial Origins of Comparative Development: An Empirical Investigation," American Economic Review, American Economic Association, vol. 91(5), pages 1369-1401, December.
    10. Michael Hout & Avery M. Guest, 2013. "Intergenerational Occupational Mobility in Great Britain and the United States since 1850: Comment," American Economic Review, American Economic Association, vol. 103(5), pages 2021-2040, August.
    11. Paul J. Ferraro & Pallavi Shukla, 2020. "Feature—Is a Replicability Crisis on the Horizon for Environmental and Resource Economics?," Review of Environmental Economics and Policy, University of Chicago Press, vol. 14(2), pages 339-351.
    12. Reinhard Selten & Thorsten Chmura, 2008. "Stationary Concepts for Experimental 2x2-Games," American Economic Review, American Economic Association, vol. 98(3), pages 938-966, June.
    13. Dewald, William G & Thursby, Jerry G & Anderson, Richard G, 1986. "Replication in Empirical Economics: The Journal of Money, Credit and Banking Project," American Economic Review, American Economic Association, vol. 76(4), pages 587-603, September.
    14. Freese, Jeremy & Peterson, David, 2017. "Replication in Social Science," SocArXiv 5bck9, Center for Open Science.
    15. Eva Vivalt, 2020. "How Much Can We Generalize From Impact Evaluations?," Journal of the European Economic Association, European Economic Association, vol. 18(6), pages 3045-3089.
    16. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    17. Michael Callen & Mohammad Isaqzadeh & James D. Long & Charles Sprenger, 2014. "Violence and Risk Preference: Experimental Evidence from Afghanistan," American Economic Review, American Economic Association, vol. 104(1), pages 123-148, January.
    18. Abel Brodeur & Scott Carrell & David Figlio & Lester Lusher, 2023. "Unpacking P-hacking and Publication Bias," American Economic Review, American Economic Association, vol. 113(11), pages 2974-3002, November.
    19. Sebastian Kranz & Peter Pütz, 2022. "Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics: Comment," American Economic Review, American Economic Association, vol. 112(9), pages 3124-3136, September.
    20. Raj Chetty & John N. Friedman & Jonah E. Rockoff, 2014. "Measuring the Impacts of Teachers II: Teacher Value-Added and Student Outcomes in Adulthood," American Economic Review, American Economic Association, vol. 104(9), pages 2633-2679, September.
    21. Francesco Caselli & Antonio Ciccone, 2019. "The Human Capital Stock: A Generalized Approach Comment," CRC TR 224 Discussion Paper Series crctr224_2019_086, University of Bonn and University of Mannheim, Germany.
    22. Benjamin F. Jones, 2014. "The Human Capital Stock: A Generalized Approach," American Economic Review, American Economic Association, vol. 104(11), pages 3752-3777, November.
    23. Whittington, Leslie A & Alm, James & Peters, H Elizabeth, 1990. "Fertility and the Personal Exemption: Implicit Pronatalist Policy in the United States," American Economic Review, American Economic Association, vol. 80(3), pages 545-556, June.
    24. Anthony C. Fisher & W. Michael Hanemann & Michael J. Roberts & Wolfram Schlenker, 2012. "The Economic Impacts of Climate Change: Evidence from Agricultural Output and Random Fluctuations in Weather: Comment," American Economic Review, American Economic Association, vol. 102(7), pages 3749-3760, December.
    25. David Y. Albouy, 2012. "The Colonial Origins of Comparative Development: An Empirical Investigation: Comment," American Economic Review, American Economic Association, vol. 102(6), pages 3059-3076, October.
    26. Brodeur, Abel & Esterling, Kevin & Ankel-Peters, Jörg & Bueno, Natália S. & Desposato, Scott & Dreber, Anna & Genovese, Federica & Green, Donald P. & Hepplewhite, Matthew & Hoces de la Guardia, Fernan, 2024. "Promoting Reproducibility and Replicability in Political Science," I4R Discussion Paper Series 100, The Institute for Replication (I4R).
    27. Daniel S. Hamermesh, 2007. "Viewpoint: Replication in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 40(3), pages 715-733, August.
    28. James Andreoni & Charles Sprenger, 2012. "Risk Preferences Are Not Time Preferences," American Economic Review, American Economic Association, vol. 102(7), pages 3357-3376, December.
    29. H. M. Collins, 1991. "The Meaning of Replication and the Science of Economics," History of Political Economy, Duke University Press, vol. 23(1), pages 123-142, Spring.
    30. Richard Crump & Gopi Shah Goda & Kevin J. Mumford, 2011. "Fertility and the Personal Exemption: Comment," American Economic Review, American Economic Association, vol. 101(4), pages 1616-1628, June.
    31. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 48(1), pages 62-83.
    32. Andrew C. Chang & Phillip Li, 2022. "Is Economics Research Replicable? Sixty Published Papers From Thirteen Journals Say “Often Notâ€," Critical Finance Review, now publishers, vol. 11(1), pages 185-206, February.
    33. Derek Lemoine & Ivan Rudik, 2017. "Steering the Climate System: Using Inertia to Lower the Cost of Policy," American Economic Review, American Economic Association, vol. 107(10), pages 2947-2957, October.
    34. Jörg Peters & Jörg Langbein & Gareth Roberts, 2018. "Generalization in the Tropics – Development Policy, Randomized Controlled Trials, and External Validity," The World Bank Research Observer, World Bank, vol. 33(1), pages 34-64.
    35. David Roodman & Jonathan Morduch, 2014. "The Impact of Microcredit on the Poor in Bangladesh: Revisiting the Evidence," Journal of Development Studies, Taylor & Francis Journals, vol. 50(4), pages 583-604, April.
    36. Francesco Caselli & Antonio Ciccone, 2019. "The Human Capital Stock: A Generalized Approach: Comment," American Economic Review, American Economic Association, vol. 109(3), pages 1155-1174, March.
    37. Jason Long & Joseph Ferrie, 2013. "Intergenerational Occupational Mobility in Great Britain and the United States since 1850," American Economic Review, American Economic Association, vol. 103(4), pages 1109-1137, June.
    38. Yu Xie & Alexandra Killewald, 2013. "Intergenerational Occupational Mobility in Great Britain and the United States since 1850: Comment," American Economic Review, American Economic Association, vol. 103(5), pages 2003-2020, August.
    39. Craig Burnside, 2011. "The Cross Section of Foreign Currency Risk Premia and Consumption Growth Risk: Comment," American Economic Review, American Economic Association, vol. 101(7), pages 3456-3476, December.
    40. David Card & Stefano DellaVigna, 2020. "What Do Editors Maximize? Evidence from Four Economics Journals," The Review of Economics and Statistics, MIT Press, vol. 102(1), pages 195-217, March.
    41. Dahal, Mahesh & Fiala, Nathan, 2020. "What do we know about the impact of microfinance? The problems of statistical power and precision," World Development, Elsevier, vol. 128(C).
    42. Jason Long & Joseph Ferrie, 2013. "Intergenerational Occupational Mobility in Great Britain and the United States since 1850: Reply," American Economic Review, American Economic Association, vol. 103(5), pages 2041-2049, August.
    43. Jesse Rothstein, 2017. "Measuring the Impacts of Teachers: Comment," American Economic Review, American Economic Association, vol. 107(6), pages 1656-1684, June.
    44. Ankel-Peters, Jörg & Fiala, Nathan & Neubauer, Florian, 2023. "Do economists replicate?," Journal of Economic Behavior & Organization, Elsevier, vol. 212(C), pages 219-232.
    45. James Berry & Lucas C. Coffman & Douglas Hanley & Rania Gihleb & Alistair J. Wilson, 2017. "Assessing the Rate of Replication in Economics," American Economic Review, American Economic Association, vol. 107(5), pages 27-31, May.
    46. Stephen L. Cheung, 2015. "Risk Preferences Are Not Time Preferences: On the Elicitation of Time Preference under Conditions of Risk: Comment," American Economic Review, American Economic Association, vol. 105(7), pages 2242-2260, July.
    47. Thomas Epper & Helga Fehr-Duda, 2015. "Risk Preferences Are Not Time Preferences: Balancing on a Budget Line: Comment," American Economic Review, American Economic Association, vol. 105(7), pages 2261-2271, July.
    48. Leamer, Edward E, 1983. "Let's Take the Con Out of Econometrics," American Economic Review, American Economic Association, vol. 73(1), pages 31-43, March.
    49. Michael A. Clemens, 2017. "The Meaning Of Failed Replications: A Review And Proposal," Journal of Economic Surveys, Wiley Blackwell, vol. 31(1), pages 326-342, February.
    50. Martín-Martín, Alberto & Orduna-Malea, Enrique & Thelwall, Mike & Delgado López-Cózar, Emilio, 2018. "Google Scholar, Web of Science, and Scopus: A systematic comparison of citations in 252 subject categories," Journal of Informetrics, Elsevier, vol. 12(4), pages 1160-1177.
    51. Lucas C. Coffman & Muriel Niederle & Alistair J. Wilson, 2017. "A Proposal to Organize and Promote Replications," American Economic Review, American Economic Association, vol. 107(5), pages 41-45, May.
    52. Sandip Sukhtankar, 2017. "Replications in Development Economics," American Economic Review, American Economic Association, vol. 107(5), pages 32-36, May.
    53. repec:ejw:journl:v:3:y:2006:i:2:p:275-282 is not listed on IDEAS
    54. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    55. repec:osf:socarx:5bck9_v1 is not listed on IDEAS
    56. Abel Brodeur & Nikolai Cook & Anthony Heyes, 2020. "Methods Matter: p-Hacking and Publication Bias in Causal Analysis in Economics," American Economic Review, American Economic Association, vol. 110(11), pages 3634-3660, November.
    57. Callen, Mike & Isaqzadeh, Mohammad & Long, James D. & Sprenger, Charles, 2014. "Violence and risk preference: experimental evidence from Afghanistan," LSE Research Online Documents on Economics 102932, London School of Economics and Political Science, LSE Library.
    58. Bin Miao & Songfa Zhong, 2015. "Risk Preferences Are Not Time Preferences: Separating Risk and Time Preference: Comment," American Economic Review, American Economic Association, vol. 105(7), pages 2272-2286, July.
    59. Owen Ozier, 2021. "Replication Redux: The Reproducibility Crisis and the Case of Deworming [Economics of Mass Deworming Programs]," The World Bank Research Observer, World Bank, vol. 36(1), pages 101-130.
    60. Daron Acemoglu & Simon Johnson & James A. Robinson & Pierre Yared, 2008. "Income and Democracy," American Economic Review, American Economic Association, vol. 98(3), pages 808-842, June.
    61. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    62. Zohid Askarov & Anthony Doucouli & Hristos Doucouli & T D Stanley, 2023. "The Significance of Data-Sharing Policy," Journal of the European Economic Association, European Economic Association, vol. 21(3), pages 1191-1226.
    63. Hensel, Przemysław G., 2021. "Reproducibility and replicability crisis: How management compares to psychology and economics – A systematic review of literature," European Management Journal, Elsevier, vol. 39(5), pages 577-594.
    64. Amir Rubin & Eran Rubin, 2021. "Systematic Bias in the Progress of Research," Journal of Political Economy, University of Chicago Press, vol. 129(9), pages 2666-2719.
    65. Linus Mattauch & H. Damon Matthews & Richard Millar & Armon Rezai & Susan Solomon & Frank Venmans, 2020. "Steering the Climate System: Using Inertia to Lower the Cost of Policy: Comment," American Economic Review, American Economic Association, vol. 110(4), pages 1231-1237, April.
    66. Daniel S. Hamermesh, 2018. "Citations in Economics: Measurement, Uses, and Impacts," Journal of Economic Literature, American Economic Association, vol. 56(1), pages 115-156, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Anna Dreber & Magnus Johannesson, 2025. "A framework for evaluating reproducibility and replicability in economics," Economic Inquiry, Western Economic Association International, vol. 63(2), pages 338-356, April.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Ankel-Peters, Jörg & Fiala, Nathan & Neubauer, Florian, 2023. "Do economists replicate?," Journal of Economic Behavior & Organization, Elsevier, vol. 212(C), pages 219-232.
    2. Anna Dreber & Magnus Johannesson, 2025. "A framework for evaluating reproducibility and replicability in economics," Economic Inquiry, Western Economic Association International, vol. 63(2), pages 338-356, April.
    3. Balafoutas, Loukas & Celse, Jeremy & Karakostas, Alexandros & Umashev, Nicholas, 2025. "Incentives and the replication crisis in social sciences: A critical review of open science practices," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 114(C).
    4. Ankel-Peters, Jörg & Fiala, Nathan & Neubauer, Florian, 2023. "Is economics self-correcting? Replications in the American Economic Review," Ruhr Economic Papers 1005, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    5. Ferman, Bruno & Finamor, Lucas, 2025. "There must be an error here! Experimental evidence on coding errors' biases," I4R Discussion Paper Series 266, The Institute for Replication (I4R).
    6. Christophe Hurlin & Jean-Edouard Colliard & Christophe Pérignon, 2024. "The Economics of Computational Reproducibility," Working Papers hal-04759240, HAL.
    7. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    8. repec:osf:osfxxx:mydzv_v1 is not listed on IDEAS
    9. Miloš Fišar & Ben Greiner & Christoph Huber & Elena Katok & Ali I. Ozkes, 2024. "Reproducibility in Management Science," Management Science, INFORMS, vol. 70(3), pages 1343-1356, March.
    10. Buchner, Martin & Rose, Julian & Johannesson, Magnus & Malan, Mandy & Ankel-Peters, Jörg, 2025. "Seeking Scientific Consensus - An Expert Survey on the Replication Debate between Acemoglu et al. (2001) and Albouy (2012)," I4R Discussion Paper Series 270, The Institute for Replication (I4R).
    11. Guillaume Coqueret, 2023. "Forking paths in financial economics," Papers 2401.08606, arXiv.org.
    12. Sebastian Gechert & Bianka Mey & Matej Opatrny & Tomas Havranek & T. D. Stanley & Pedro R. D. Bom & Hristos Doucouliagos & Philipp Heimberger & Zuzana Irsova & Heiko J. Rachinger, 2025. "Conventional wisdom, meta‐analysis, and research revision in economics," Journal of Economic Surveys, Wiley Blackwell, vol. 39(3), pages 980-999, July.
    13. Sylvérie Herbert & Hautahi Kingi & Flavio Stanchi & Lars Vilhubern, 2021. "The Reproducibility of Economics Research: A Case Study," Working papers 853, Banque de France.
    14. Lukas Fink & Jan Marcus, 2025. "Replication code availability over time and across fields: Evidence from the German Socio‐Economic Panel," Economic Inquiry, Western Economic Association International, vol. 63(2), pages 357-386, April.
    15. Sylvérie Herbert & Hautahi Kingi & Flavio Stanchi & Lars Vilhuber, 2024. "Reproduce to validate: A comprehensive study on the reproducibility of economics research," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 57(3), pages 961-988, August.
    16. Thibaut Arpinon & Marianne Lefebvre, 2024. "Registered Reports and Associated Benefits for Agricultural Economics," Post-Print hal-04635986, HAL.
    17. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 48(1), pages 62-83.
    18. Hensel, Przemysław G., 2021. "Reproducibility and replicability crisis: How management compares to psychology and economics – A systematic review of literature," European Management Journal, Elsevier, vol. 39(5), pages 577-594.
    19. Klege, Rebecca A. & Amuakwa-Mensah, Franklin & Visser, Martine, 2022. "Tenancy and energy choices in Rwanda. A replication and extension study," World Development Perspectives, Elsevier, vol. 26(C).
    20. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    21. Rose, Julian & Neubauer, Florian & Ankel-Peters, Jörg, 2024. "Long-Term Effects of the Targeting the Ultra-Poor Program - A Reproducibility and Replicability Assessment of Banerjee et al. (2021)," I4R Discussion Paper Series 142, The Institute for Replication (I4R).

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:ecinqu:v:63:y:2025:i:2:p:463-485. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/weaaaea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.