IDEAS home Printed from https://ideas.repec.org/p/zbw/rwirep/1005.html
   My bibliography  Save this paper

Is economics self-correcting? Replications in the American Economic Review

Author

Listed:
  • Ankel-Peters, Jörg
  • Fiala, Nathan
  • Neubauer, Florian

Abstract

Replication and constructive controversy are essential for scientific progress. This paper reviews the impact of all replications published as comments in the American Economic Review between 2010 and 2020. We investigate the citation rates of comments and whether a comment affects its original paper's citation rates. We find that most comments are barely cited, and they have no impact on the original papers' subsequent citations. This finding holds for original papers for which the comment diagnoses a substantive problem. We conclude from these citation patterns that replications do not update the economics literature. In an online opinion survey, we elicited viewpoints of both comment authors and original authors and find that in most cases, there is no consensus regarding the replication's success and to what extent the original paper's contribution sustains. This resonates with the conventional wisdom that robustness and replicability are hard to define in economics.

Suggested Citation

  • Ankel-Peters, Jörg & Fiala, Nathan & Neubauer, Florian, 2023. "Is economics self-correcting? Replications in the American Economic Review," Ruhr Economic Papers 1005, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    • Handle: RePEc:zbw:rwirep:1005
    DOI: 10.4419/96973171
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/269235/1/1838894071.pdf
    Download Restriction: no
    File URL: https://libkey.io/10.4419/96973171?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. David Y. Albouy, 2012. "The Colonial Origins of Comparative Development: An Empirical Investigation: Comment," American Economic Review, American Economic Association, vol. 102(6), pages 3059-3076, October.
    2. Daron Acemoglu & Simon Johnson & James A. Robinson & Pierre Yared, 2008. "Income and Democracy," American Economic Review, American Economic Association, vol. 98(3), pages 808-842, June.
    3. Lucas C. Coffman & Muriel Niederle & Alistair J. Wilson, 2017. "A Proposal to Organize and Promote Replications," American Economic Review, American Economic Association, vol. 107(5), pages 41-45, May.
    4. Derek Lemoine & Ivan Rudik, 2017. "Steering the Climate System: Using Inertia to Lower the Cost of Policy," American Economic Review, American Economic Association, vol. 107(10), pages 2947-2957, October.
    5. H. M. Collins, 1991. "The Meaning of Replication and the Science of Economics," History of Political Economy, Duke University Press, vol. 23(1), pages 123-142, Spring.
    6. Jason Long & Joseph Ferrie, 2013. "Intergenerational Occupational Mobility in Great Britain and the United States since 1850: Reply," American Economic Review, American Economic Association, vol. 103(5), pages 2041-2049, August.
    7. Daniel S. Hamermesh, 2007. "Replication in Economics," NBER Working Papers 13026, National Bureau of Economic Research, Inc.
    8. Daron Acemoglu & Simon Johnson & James A. Robinson, 2001. "The Colonial Origins of Comparative Development: An Empirical Investigation," American Economic Review, American Economic Association, vol. 91(5), pages 1369-1401, December.
    9. Michael Hout & Avery M. Guest, 2013. "Intergenerational Occupational Mobility in Great Britain and the United States since 1850: Comment," American Economic Review, American Economic Association, vol. 103(5), pages 2021-2040, August.
    10. Whittington, Leslie A & Alm, James & Peters, H Elizabeth, 1990. "Fertility and the Personal Exemption: Implicit Pronatalist Policy in the United States," American Economic Review, American Economic Association, vol. 80(3), pages 545-556, June.
    11. Richard Crump & Gopi Shah Goda & Kevin J. Mumford, 2011. "Fertility and the Personal Exemption: Comment," American Economic Review, American Economic Association, vol. 101(4), pages 1616-1628, June.
    12. Ankel-Peters, Jörg & Fiala, Nathan & Neubauer, Florian, 2023. "Do economists replicate?," Journal of Economic Behavior & Organization, Elsevier, vol. 212(C), pages 219-232.
    13. Daniel S. Hamermesh, 2007. "Viewpoint: Replication in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 40(3), pages 715-733, August.
    14. Jason Long & Joseph Ferrie, 2013. "Intergenerational Occupational Mobility in Great Britain and the United States since 1850," American Economic Review, American Economic Association, vol. 103(4), pages 1109-1137, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Rose, Julian & Neubauer, Florian & Ankel-Peters, Jörg, 2024. "Long-term effects of the targeting the ultra-poor program: A reproducibility and replicability assessment of Banerjee et al. (2021)," Ruhr Economic Papers 1107, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    2. Ankel-Peters, Jörg & Fiala, Nathan & Neubauer, Florian, 2023. "Do economists replicate?," Journal of Economic Behavior & Organization, Elsevier, vol. 212(C), pages 219-232.
    3. Tom Coupé & Thomas Logchies & W. Robert Reed, 2024. "Do Replications Really Receive Fewer Citations?," Working Papers in Economics 24/15, University of Canterbury, Department of Economics and Finance.
    4. Ankel-Peters, Jörg & Schmidt, Christoph M., 2023. "Rural electrification, the credibility revolution, and the limits of evidence-based policy," Ruhr Economic Papers 1051, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    5. Thibaut Arpinon & Marianne Lefebvre, 2024. "Registered Reports and Associated Benefits for Agricultural Economics," Post-Print hal-04635986, HAL.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Maren Duvendack & Richard Palmer-Jones, 2013. "Replication of quantitative work in development studies: Experiences and suggestions," Progress in Development Studies, , vol. 13(4), pages 307-322, October.
    2. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    3. Luigi Butera & Philip Grossman & Daniel Houser & John List & Marie-Claire Villeval, 2020. "A New Mechanism to Alleviate the Crises of Confidence in Science - With an Application to the Public Goods Game," Artefactual Field Experiments 00684, The Field Experiments Website.
    4. Casey, Gregory & Klemp, Marc, 2016. "Instrumental Variables in the Long Run," MPRA Paper 68696, University Library of Munich, Germany.
    5. Markus Jäntti & Stephen P. Jenkins, 2013. "Income Mobility," SOEPpapers on Multidisciplinary Panel Data Research 607, DIW Berlin, The German Socio-Economic Panel (SOEP).
    6. David S. Johnson & Catherine Massey & Amy O’Hara, 2015. "The Opportunities and Challenges of Using Administrative Data Linkages to Evaluate Mobility," The ANNALS of the American Academy of Political and Social Science, , vol. 657(1), pages 247-264, January.
    7. Miloš Fišar & Ben Greiner & Christoph Huber & Elena Katok & Ali I. Ozkes, 2024. "Reproducibility in Management Science," Management Science, INFORMS, vol. 70(3), pages 1343-1356, March.
    8. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    9. Michael A. Clemens, 2017. "The Meaning Of Failed Replications: A Review And Proposal," Journal of Economic Surveys, Wiley Blackwell, vol. 31(1), pages 326-342, February.
    10. Saima Nawaz & M. Idrees Khawaja, 2016. "Fiscal Policy, Institutions And Growth: New Insights," The Singapore Economic Review (SER), World Scientific Publishing Co. Pte. Ltd., vol. 64(05), pages 1251-1278, December.
    11. Ankel-Peters, Jörg & Fiala, Nathan & Neubauer, Florian, 2023. "Do economists replicate?," Journal of Economic Behavior & Organization, Elsevier, vol. 212(C), pages 219-232.
    12. Zachary Ward, 2023. "Intergenerational Mobility in American History: Accounting for Race and Measurement Error," American Economic Review, American Economic Association, vol. 113(12), pages 3213-3248, December.
    13. Zhang, Yinjunjie & Hoffmann, Manuel & Sara, Raisa & Eckel, Catherine, 2024. "Fairness preferences revisited," Journal of Economic Behavior & Organization, Elsevier, vol. 223(C), pages 278-306.
    14. Mußhoff, Oliver & Ölkers, Tim & Kirchner, Ella, 2024. "A note on the discussion regarding terrorism and land use in agriculture," Land Use Policy, Elsevier, vol. 144(C).
    15. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," Research Policy, Elsevier, vol. 48(1), pages 62-83.
    16. Sylvérie Herbert & Hautahi Kingi & Flavio Stanchi & Lars Vilhubern, 2021. "The Reproducibility of Economics Research: A Case Study," Working papers 853, Banque de France.
    17. Sen, Kunal & Pritchett, Lant & Kar, Sabyasachi & Raihan, Selim, 2016. "Democracy Versus Dictatorship? The Political Determinants of Growth Episodes," Working Paper Series rwp17-009, Harvard University, John F. Kennedy School of Government.
    18. Roy Chen & Yan Chen & Yohanes E. Riyanto, 2021. "Best practices in replication: a case study of common information in coordination games," Experimental Economics, Springer;Economic Science Association, vol. 24(1), pages 2-30, March.
    19. Mark J. McCabe & Frank Mueller-Langer, 2019. "Does Data Disclosure Increase Citations? Empirical Evidence from a Natural Experiment in Leading Economics Journals," JRC Working Papers on Digital Economy 2019-02, Joint Research Centre.
    20. Natkhov, T. & Polishchuk, L., 2017. "Political Economy of Institutions and Development: The Importance of Being Inclusive. Reflection on "Why Nations Fail" by D. Acemoglu and J. Robinson. Part I. Institutions and Economic Devel," Journal of the New Economic Association, New Economic Association, vol. 34(2), pages 12-38.

    More about this item

    Keywords

    Replication; citations; meta-science;
    All these keywords.

    JEL classification:

    • A11 - General Economics and Teaching - - General Economics - - - Role of Economics; Role of Economists
    • A14 - General Economics and Teaching - - General Economics - - - Sociology of Economics

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:rwirep:1005. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/rwiesde.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.