IDEAS home Printed from https://ideas.repec.org/p/cbt/econwp/16-34.html
   My bibliography  Save this paper

What is Meant by ‘Replication’ and Why Does It Encounter Resistance in Economics?

Author

Listed:

Abstract

This paper discusses recent trends in the use of replications in economics. We identify a number of sources of progress, including the results of recent replication studies that have attempted to identify replication rates within the discipline. These studies generally find that replication rates are relatively low, though they may be higher for laboratory experiments in economics. We also identify two web-based resources for replications, the Replication in Economics wiki and The Replication Network. We then consider obstacles to undertaking replication studies in economics. Two obstacles are the lack of publishing outlets and difficulties in obtaining data and code for published studies. We identify journals that publish replication studies and that “regularly” include data and code as supplementary files for their published research. Finally, we highlight replication initiatives in psychology and political science, behind which economics appears to lag. Whether this is because the problems that beset those disciplines are less severe in economics, or because economics is more resistant to replications, is arguable.

Suggested Citation

  • Maren Duvendack & Richard Palmer-Jones & W. Robert Reed, 2016. "What is Meant by ‘Replication’ and Why Does It Encounter Resistance in Economics?," Working Papers in Economics 16/34, University of Canterbury, Department of Economics and Finance.
  • Handle: RePEc:cbt:econwp:16/34
    as

    Download full text from publisher

    File URL: https://repec.canterbury.ac.nz/cbt/econwp/1634.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Daniel S. Hamermesh, 2007. "Viewpoint: Replication in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 40(3), pages 715-733, August.
    2. Michael A. Clemens, 2017. "The Meaning Of Failed Replications: A Review And Proposal," Journal of Economic Surveys, Wiley Blackwell, vol. 31(1), pages 326-342, February.
    3. Vlaeminck, Sven & Herrmann, Lisa-Kristin, 2015. "Data Policies and Data Archives: A New Paradigm for Academic Publishing in Economic Sciences?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, pages 145-155.
    4. B.D. McCullough, 2009. "Open Access Economics Journals and the Market for Reproducible Economic Research," Economic Analysis and Policy, Elsevier, vol. 39(1), pages 117-126, March.
    5. Vlaeminck, Sven, 2013. "Data Management in Scholarly Journals and Possible Roles for Libraries - Some Insights from EDaWaX," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, pages 49-79.
    6. Maren Duvendack & Richard W. Palmer-Jones & W. Robert Reed, 2015. "Replications in Economics: A Progress Report," Econ Journal Watch, Econ Journal Watch, vol. 12(2), pages 164–191-1, May.
    7. Hamermesh, Daniel S., 2007. "Replication in Economics," IZA Discussion Papers 2760, Institute of Labor Economics (IZA).
    8. Michael Clemens, 2015. "The Meaning of Failed Replications: A Review and Proposal - Working Paper 399," Working Papers 399, Center for Global Development.
    9. Maren Duvendack & Richard Palmer-Jones, 2013. "Replication of quantitative work in development studies: Experiences and suggestions," Progress in Development Studies, , vol. 13(4), pages 307-322, October.
    10. Andrew C. Chang & Phillip Li, 2015. "Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say \"Usually Not\"," Finance and Economics Discussion Series 2015-83, Board of Governors of the Federal Reserve System (U.S.).
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Hernández Alemán, Anastasia & León, Carmelo J., 2018. "La Réplica en el Análisis Económico Aplicado/Replication in Applied Economic Analysis," Estudios de Economia Aplicada, Estudios de Economia Aplicada, vol. 36, pages 317-332, Enero.
    2. Hardcopf, Rick & Liu, Gensheng (Jason) & Shah, Rachna, 2021. "Lean production and operational performance: The influence of organizational culture," International Journal of Production Economics, Elsevier, vol. 235(C).
    3. Hubbard, Raymond, 2017. "A proposal for replicating Evanschitzky, Baumgarth, Hubbard, and Armstrong's "Replication research's disturbing trend" (Journal of Business Research, 2007)," Economics Discussion Papers 2017-75, Kiel Institute for the World Economy (IfW).
    4. Romain Espinosa & Anis Nassar, 2021. "The Acceptability of Food Policies," Post-Print halshs-03210654, HAL.
    5. Valérie Orozco & Christophe Bontemps & Elise Maigne & Virginie Piguet & Annie Hofstetter & Anne Marie Lacroix & Fabrice Levert & Jean-Marc Rousselle, 2017. "How to make a pie? Reproducible Research for Empirical Economics & Econometrics," Post-Print hal-01939942, HAL.
    6. Emma McManus & David Turner & Tracey Sach, 2019. "Can You Repeat That? Exploring the Definition of a Successful Model Replication in Health Economics," PharmacoEconomics, Springer, vol. 37(11), pages 1371-1381, November.
    7. Bull, Charles & Courty, Pascal & Doyon, Maurice & Rondeau, Daniel, 2019. "Failure of the Becker–DeGroot–Marschak mechanism in inexperienced subjects: New tests of the game form misconception hypothesis," Journal of Economic Behavior & Organization, Elsevier, vol. 159(C), pages 235-253.
    8. Heiko T. Burret & Lars P. Feld & Christoph A. Schaltegger, 2018. "Fiscal Federalism and Economic Performance - New Evidence from Switzerland," CESifo Working Paper Series 7250, CESifo.
    9. Owen, P. Dorian, 2018. "Replication to assess statistical adequacy," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW), vol. 12, pages 1-16.
    10. Daniels, Gerald Eric & Kakar, Venoo, 2018. "Normalized CES supply-side system approach: How to replicate Klump, McAdam, and Willman (2007)," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW), vol. 12, pages 1-11.
    11. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    12. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    13. Anderson, Richard G., 2017. "Should you choose to do so...: A replication paradigm," Economics Discussion Papers 2017-79, Kiel Institute for the World Economy (IfW).
    14. Daniels, Gerald Eric & Kakar, Venoo, 2017. "Normalized CES supply-side system approach: How to replicate Klump, McAdam, and Willman (Review of Economics and Statistics, 2007)," Economics Discussion Papers 2017-70, Kiel Institute for the World Economy (IfW).
    15. Josephson, Anna & Michler, Jeffrey D., 2018. "Viewpoint: Beasts of the field? Ethics in agricultural and applied economics," Food Policy, Elsevier, vol. 79(C), pages 1-11.
    16. Stephanie Rosch & Sharon Raszap Skorbiansky & Collin Weigel & Kent D. Messer & Daniel Hellerstein, 2021. "Barriers to Using Economic Experiments in Evidence‐Based Agricultural Policymaking," Applied Economic Perspectives and Policy, John Wiley & Sons, vol. 43(2), pages 531-555, June.
    17. Lucas C. Coffman & Muriel Niederle & Alistair J. Wilson, 2017. "A Proposal to Organize and Promote Replications," American Economic Review, American Economic Association, vol. 107(5), pages 41-45, May.
    18. Hannum, Randall J., 2017. "A replication plan for "Does social media reduce corruption?" (Information Economics and Policy, 2017)," Economics Discussion Papers 2017-72, Kiel Institute for the World Economy (IfW).
    19. Brade, Raphael & Himmler, Oliver & Jäckle, Robert, 2018. "Normatively Framed Relative Performance Feedback – Field Experiment and Replication," MPRA Paper 88830, University Library of Munich, Germany.
    20. Bédécarrats, Florent & Guérin, Isabelle & Morvant-Roux, Solène & Roubaud, François, 2019. "Estimating microcredit impact with low take-up, contamination and inconsistent data. A replication study of Crépon, Devoto, Duflo, and Parienté (American Economic Journal: Applied Economics, 2015)," International Journal for Re-Views in Empirical Economics (IREE), ZBW - Leibniz Information Centre for Economics, vol. 3(2019-3), pages 1-22.
    21. Florent Bédécarrats & Isabelle Guérin & Solène Morvant-Roux & François Roubaud, 2018. "Verifying the internal validity of a flagship RCT: A review of Crépon, Devoto, Duflo and Pariente (American Economic Journal: Applied Economics, 2015)," Working Papers DT/2018/09, DIAL (Développement, Institutions et Mondialisation).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Nicolas Vallois & Dorian Jullien, 2017. "Replication in experimental economics: A historical and quantitative approach focused on public good game experiments," Working Papers halshs-01651080, HAL.
    2. Nicolas Vallois & Dorian Jullien, 2017. "Replication in experimental economics: A historical and quantitative approach focused on public good game experiments," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) halshs-01651080, HAL.
    3. Christophe Hurlin & Christophe Pérignon, 2020. "Reproducibility Certification in Economics Research," Working Papers hal-02896404, HAL.
    4. Valérie Orozco & Christophe Bontemps & Elise Maigne & Virginie Piguet & Annie Hofstetter & Anne Marie Lacroix & Fabrice Levert & Jean-Marc Rousselle, 2017. "How to make a pie? Reproducible Research for Empirical Economics & Econometrics," Post-Print hal-01939942, HAL.
    5. Nicolas Vallois & Dorian Jullien, 2017. "Replication in Experimental Economics: A Historical and Quantitative Approach Focused on Public Good Game Experiments," GREDEG Working Papers 2017-21, Groupe de REcherche en Droit, Economie, Gestion (GREDEG CNRS), Université Côte d'Azur, France.
    6. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, pages 62-83.
    7. Grolleau, Gilles & Ibanez, Lisette & Mzoughi, Naoufel, 2020. "Moral judgment of environmental harm caused by a single versus multiple wrongdoers: A survey experiment," Ecological Economics, Elsevier, vol. 170(C).
    8. Chin, Jason & Zeiler, Kathryn, 2021. "Replicability in Empirical Legal Research," LawArXiv 2b5k4, Center for Open Science.
    9. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    10. Hernández Alemán, Anastasia & León, Carmelo J., 2018. "La Réplica en el Análisis Económico Aplicado/Replication in Applied Economic Analysis," Estudios de Economia Aplicada, Estudios de Economia Aplicada, vol. 36, pages 317-332, Enero.
    11. Koten, Silvester Van, 2020. "Forward premia in electricity markets: A replication study," Energy Economics, Elsevier, vol. 89(C).
    12. Daniels, Gerald Eric & Kakar, Venoo, 2017. "Normalized CES supply-side system approach: How to replicate Klump, McAdam, and Willman (Review of Economics and Statistics, 2007)," Economics Discussion Papers 2017-70, Kiel Institute for the World Economy (IfW).
    13. Daniels, Gerald Eric & Kakar, Venoo, 2018. "Normalized CES supply-side system approach: How to replicate Klump, McAdam, and Willman (2007)," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW), vol. 12, pages 1-11.
    14. Opoku-Agyemang, Kweku A., 2017. "A Human-Computer Interaction Approach for Integrity in Economics," SocArXiv ra3cs, Center for Open Science.
    15. Vlaeminck, Sven & Podkrajac, Felix, 2017. "Journals in Economic Sciences: Paying Lip Service to Reproducible Research?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, pages 1-16.
    16. Hussinger, Katrin & Pellens, Maikel, 2019. "Guilt by association: How scientific misconduct harms prior collaborators," Research Policy, Elsevier, vol. 48(2), pages 516-530.
    17. Nick Huntington‐Klein & Andreu Arenas & Emily Beam & Marco Bertoni & Jeffrey R. Bloem & Pralhad Burli & Naibin Chen & Paul Grieco & Godwin Ekpe & Todd Pugatch & Martin Saavedra & Yaniv Stopnitzky, 2021. "The influence of hidden researcher decisions in applied microeconomics," Economic Inquiry, Western Economic Association International, vol. 59(3), pages 944-960, July.
    18. Roy Chen & Yan Chen & Yohanes E. Riyanto, 2021. "Best practices in replication: a case study of common information in coordination games," Experimental Economics, Springer;Economic Science Association, vol. 24(1), pages 2-30, March.
    19. Bruns, Stephan B. & König, Johannes & Stern, David I., 2019. "Replication and robustness analysis of ‘energy and economic growth in the USA: A multivariate approach’," Energy Economics, Elsevier, vol. 82(C), pages 100-113.
    20. Kaplan, Greg & Mitman, Kurt & Violante, Giovanni L., 2020. "Non-durable consumption and housing net worth in the Great Recession: Evidence from easily accessible data," Journal of Public Economics, Elsevier, vol. 189(C).

    More about this item

    Keywords

    Replication; data sharing; publication bias;
    All these keywords.

    JEL classification:

    • A1 - General Economics and Teaching - - General Economics
    • B4 - Schools of Economic Thought and Methodology - - Economic Methodology

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cbt:econwp:16/34. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/decannz.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Albert Yee (email available below). General contact details of provider: https://edirc.repec.org/data/decannz.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.