IDEAS home Printed from https://ideas.repec.org/a/bla/ecinqu/v63y2025i2p338-356.html

A framework for evaluating reproducibility and replicability in economics

Author

Listed:
  • Anna Dreber
  • Magnus Johannesson

Abstract

We propose a framework for evaluating reproducibility and replicability in economics. Reproducibility is defined as testing if the results of an original study can be reproduced using the same data and replicability is defined as testing if the results of an original study hold in new data. We further divide reproducibility and replicability studies into five types: computational reproducibility, recreate reproducibility, robustness reproducibility, direct replicability and conceptual replicability. In addition to this typology we propose indicators to measure the degree of reproducibility and replicability in both individual studies and for a group of studies.

Suggested Citation

  • Anna Dreber & Magnus Johannesson, 2025. "A framework for evaluating reproducibility and replicability in economics," Economic Inquiry, Western Economic Association International, vol. 63(2), pages 338-356, April.
    • Handle: RePEc:bla:ecinqu:v:63:y:2025:i:2:p:338-356
    DOI: 10.1111/ecin.13244
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/ecin.13244
    Download Restriction: no
    File URL: https://libkey.io/10.1111/ecin.13244?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Baldwin, Grant & Becker, Clayton & Ortiz, Emily & Goetz, Josh, 2024. "A comment on "The people think what I think" by Furnas and LaPira (2024)," I4R Discussion Paper Series 171, The Institute for Replication (I4R), revised 2024.
    2. Enzinger, Matthias & Gechert, Sebastian & Heimberger, Philipp & Prante, Franz & Romero, Daniel F., 2025. "The overstated effects of conventional monetary policy on output and prices," I4R Discussion Paper Series 264, The Institute for Replication (I4R).
    3. McWay, Ryan & Braaksma, Matthew, 2025. "The Political Consequences of Resource Scarcity: Targeted Spending in a Water-Stressed Democracy. A Replication Study of Mahadevan and Shenoy," I4R Discussion Paper Series 231, The Institute for Replication (I4R).
    4. Bensch, Gunther & Rose, Julian & Brodeur, Abel & Ankel-Peters, Jörg, 2025. "The Robustness Dashboard," I4R Discussion Paper Series 234, The Institute for Replication (I4R).
      • Bensch, Gunther & Rose, Julian & Brodeur, Abel & Ankel-Peters, Jörg, 2025. "The robustness dashboard," Ruhr Economic Papers 1167, RWI - Leibniz-Institut für Wirtschaftsforschung, Ruhr-University Bochum, TU Dortmund University, University of Duisburg-Essen.
    5. Clerc, Melchior & Gosselin-Pali, Adrien & Wendling, Eliot, 2024. "A Replication of Macchi (2023): "Worth Your Weight: Experimental Evidence on the Benefits of Obesity in Low-Income Countries"," I4R Discussion Paper Series 145, The Institute for Replication (I4R).
    6. McWay, Ryan, 2025. "Unintended Consequences of Lockdowns, COVID-19 and the Shadow Pandemic in India. A Reproduction Study of Ravindran and Shah," I4R Discussion Paper Series 230, The Institute for Replication (I4R).
    7. Miloš Fišar & Ben Greiner & Christoph Huber & Elena Katok & Ali I. Ozkes, 2024. "Reproducibility in Management Science," Management Science, INFORMS, vol. 70(3), pages 1343-1356, March.
    8. Balafoutas, Loukas & Celse, Jeremy & Karakostas, Alexandros & Umashev, Nicholas, 2025. "Incentives and the replication crisis in social sciences: A critical review of open science practices," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 114(C).
    9. Philipp Knöpfle & Tim Schatto-Eckrodt, 2024. "The Challenges of Replicating Volatile Platform-Data Studies: Replicating Schatto-Eckrodt et al. (2020)," Media and Communication, Cogitatio Press, vol. 12.
    10. repec:osf:osfxxx:mydzv_v1 is not listed on IDEAS
    11. Marcus, Jan, 2025. "Replication code as a cornerstone of the credibility revolution 2.0," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 117(C).
    12. McWay, Ryan & Braaksma, Matthew, 2025. "The political consequences of resource scarcity: Targeted spending in a water-stressed democracy. A replication study of Mahadevan and Shenoy (Journal of Public Economics, 2023)," World Development Perspectives, Elsevier, vol. 39(C).
    13. Casas, Andreu & Dagher, Georgia & O'Loughlin, Ben, 2025. "Academic Access to Social Media Data for the Study of Political Online Safety," SocArXiv 7pcjd, Center for Open Science.

    More about this item

    JEL classification:

    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • C90 - Mathematical and Quantitative Methods - - Design of Experiments - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:ecinqu:v:63:y:2025:i:2:p:338-356. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/weaaaea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.