IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp9000.html
   My bibliography  Save this paper

The Meaning of Failed Replications: A Review and Proposal

Author

Listed:
  • Clemens, Michael A.

    () (Center for Global Development)

Abstract

The welcome rise of replication tests in economics has not been accompanied by a single, clear definition of replication. A discrepant replication, in current usage of the term, can signal anything from an unremarkable disagreement over methods to scientific incompetence or misconduct. This paper proposes an unambiguous definition of replication, one that reflects currently common but unstandardized use. It contrasts this definition with decades of unsuccessful attempts to standardize terminology, and argues that many prominent results described as replication tests – in labor, development, and other fields of economics – should not be described as such. Adopting this definition can improve incentives for researchers, encouraging more and better replication tests.

Suggested Citation

  • Clemens, Michael A., 2015. "The Meaning of Failed Replications: A Review and Proposal," IZA Discussion Papers 9000, Institute for the Study of Labor (IZA).
  • Handle: RePEc:iza:izadps:dp9000
    as

    Download full text from publisher

    File URL: http://ftp.iza.org/dp9000.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Harrison, Glenn W, 1998. "Mortgage Lending in Boston: A Reconsideration of the Evidence," Economic Inquiry, Western Economic Association International, vol. 36(1), pages 29-38, January.
    2. Levitt, Steven D, 1997. "Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime," American Economic Review, American Economic Association, vol. 87(3), pages 270-290, June.
    3. Hubbard, Raymond & Vetter, Daniel E., 1996. "An empirical comparison of published replication research in accounting, economics, finance, management, and marketing," Journal of Business Research, Elsevier, vol. 35(2), pages 153-164, February.
    4. Furman, Jeffrey L. & Jensen, Kyle & Murray, Fiona, 2012. "Governing knowledge in the scientific community: Exploring the role of retractions in biomedicine," Research Policy, Elsevier, vol. 41(2), pages 276-290.
    5. Thomas Herndon & Michael Ash & Robert Pollin, 2014. "Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff," Cambridge Journal of Economics, Oxford University Press, vol. 38(2), pages 257-279.
    6. Richard Anderson & William Greene & B. D. McCullough & H. D. Vinod, 2008. "The role of data/code archives in the future of economic research," Journal of Economic Methodology, Taylor & Francis Journals, vol. 15(1), pages 99-119.
    7. William Easterly & Ross Levine & David Roodman, 2004. "Aid, Policies, and Growth: Comment," American Economic Review, American Economic Association, vol. 94(3), pages 774-780, June.
    8. Hamermesh, Daniel S., 1997. "Some thoughts on replications and reviews," Labour Economics, Elsevier, vol. 4(2), pages 107-109, June.
    9. Daron Acemoglu & Simon Johnson & James A. Robinson, 2001. "The Colonial Origins of Comparative Development: An Empirical Investigation," American Economic Review, American Economic Association, vol. 91(5), pages 1369-1401, December.
    10. Johnson, Simon & Larson, William & Papageorgiou, Chris & Subramanian, Arvind, 2013. "Is newer better? Penn World Table Revisions and their impact on growth estimates," Journal of Monetary Economics, Elsevier, vol. 60(2), pages 255-274.
    11. John R. Lott & John Whitley, 2007. "Abortion And Crime: Unwanted Children And Out-Of-Wedlock Births," Economic Inquiry, Western Economic Association International, vol. 45(2), pages 304-324, April.
    12. Caroline M. Hoxby, 2007. "Does Competition Among Public Schools Benefit Students and Taxpayers? Reply," American Economic Review, American Economic Association, vol. 97(5), pages 2038-2055, December.
    13. Leimer, Dean R & Lesnoy, Selig D, 1982. "Social Security and Private Saving: New Time-Series Evidence," Journal of Political Economy, University of Chicago Press, vol. 90(3), pages 606-629, June.
    14. Evanschitzky, Heiner & Baumgarth, Carsten & Hubbard, Raymond & Armstrong, J. Scott, 2007. "Replication research's disturbing trend," Journal of Business Research, Elsevier, vol. 60(4), pages 411-415, April.
    15. Hatice Balli & Bent Sørensen, 2013. "Interaction effects in econometrics," Empirical Economics, Springer, vol. 45(1), pages 583-603, August.
    16. Annette N. Brown & Drew B. Cameron & Benjamin D. K. Wood, 2014. "Quality evidence for policymaking: I'll believe it when I see the replication," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(3), pages 215-235, September.
    17. Ming-Jen Lin & Ming-Ching Luoh, 2008. "Can Hepatitis B Mothers Account for the Number of Missing Women? Evidence from Three Million Newborns in Taiwan," American Economic Review, American Economic Association, vol. 98(5), pages 2259-2273, December.
    18. Jesse Rothstein, 2007. "Does Competition Among Public Schools Benefit Students and Taxpayers? Comment," American Economic Review, American Economic Association, vol. 97(5), pages 2026-2037, December.
    19. Ted Joyce, 2004. "Did Legalized Abortion Lower Crime?," Journal of Human Resources, University of Wisconsin Press, vol. 39(1).
    20. Martha J. Bailey, 2006. "More Power to the Pill: The Impact of Contraceptive Freedom on Women's Life Cycle Labor Supply," The Quarterly Journal of Economics, Oxford University Press, vol. 121(1), pages 289-320.
    21. Nancy Cartwright, 1991. "Replicability, Reproducibility, and Robustness: Comments on Harry Collins," History of Political Economy, Duke University Press, vol. 23(1), pages 143-155, Spring.
    22. Justin McCrary, 2002. "Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime: Comment," American Economic Review, American Economic Association, vol. 92(4), pages 1236-1243, September.
    23. Bruce Chapman & Yvonne Dunlop & Matthew Gray & Amy Liu & Deborah Mitchell, 2001. "The Impact of Children on the Lifetime Earnings of Australian Women: Evidence from the 1990s," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 34(4), pages 373-389.
    24. Munnell, Alicia H. & Geoffrey M. B. Tootell & Lynn E. Browne & James McEneaney, 1996. "Mortgage Lending in Boston: Interpreting HMDA Data," American Economic Review, American Economic Association, vol. 86(1), pages 25-53, March.
    25. Edward Miguel & Michael Kremer, 2004. "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities," Econometrica, Econometric Society, vol. 72(1), pages 159-217, January.
    26. Feldstein, Martin S, 1974. "Social Security, Induced Retirement, and Aggregate Capital Accumulation," Journal of Political Economy, University of Chicago Press, vol. 82(5), pages 905-926, Sept./Oct.
    27. Farley, John U & Lehmann, Donald R & Ryan, Michael J, 1981. "Generalizing from "Imperfect" Replication," The Journal of Business, University of Chicago Press, vol. 54(4), pages 597-610, October.
    28. repec:cup:apsrev:v:94:y:2000:i:02:p:305-321_22 is not listed on IDEAS
    29. Boone, Peter, 1996. "Politics and the effectiveness of foreign aid," European Economic Review, Elsevier, vol. 40(2), pages 289-329, February.
    30. B.D. McCullough & Kerry Anne McGeary & Teresa D. Harrison, 2008. "Do economics journal archives promote replicable research?," Canadian Journal of Economics, Canadian Economics Association, vol. 41(4), pages 1406-1420, November.
    31. Summers, Lawrence H, 1991. " The Scientific Illusion in Empirical Macroeconomics," Scandinavian Journal of Economics, Wiley Blackwell, vol. 93(2), pages 129-148.
    32. Graham A. Davis, 2013. "Replicating Sachs and Warner's Working Papers on the Resource Curse," Journal of Development Studies, Taylor & Francis Journals, vol. 49(12), pages 1615-1630, December.
    33. Hunter, John E, 2001. " The Desperate Need for Replications," Journal of Consumer Research, Oxford University Press, vol. 28(1), pages 149-158, June.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Nicolas Vallois & Dorian Jullien, 2017. "Replication in Experimental Economics: A Historical and Quantitative Approach Focused on Public Good Game Experiments," GREDEG Working Papers 2017-21, Groupe de REcherche en Droit, Economie, Gestion (GREDEG CNRS), University of Nice Sophia Antipolis.
    2. Latré, Edwin & Perko, Tanja & Thijssen, Peter, 2017. "Public opinion change after the Fukushima nuclear accident: The role of national context revisited," Energy Policy, Elsevier, vol. 104(C), pages 124-133.
    3. Maren Duvendack & Richard Palmer-Jones & W. Robert Reed, 2017. "What Is Meant by "Replication" and Why Does It Encounter Resistance in Economics?," American Economic Review, American Economic Association, pages 46-51.
    4. Isaiah Andrews & Maximilian Kasy, 2017. "Identification of and Correction for Publication Bias," NBER Working Papers 23298, National Bureau of Economic Research, Inc.
    5. Daniels, Gerald Eric & Kakar, Venoo, 2017. "Normalized CES supply-side system approach: How to replicate Klump, McAdam, and Willman (Review of Economics and Statistics, 2007)," Economics Discussion Papers 2017-70, Kiel Institute for the World Economy (IfW).
    6. Sebastian Galiani & Paul Gertler & Mauricio Romero, 2017. "Incentives for Replication in Economics," NBER Working Papers 23576, National Bureau of Economic Research, Inc.
    7. repec:aea:aecrev:v:107:y:2017:i:5:p:27-31 is not listed on IDEAS
    8. Nicolas Vallois & Dorian Jullien, 2017. "Replication in experimental economics: A historical and quantitative approach focused on public good game experiments," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) halshs-01651080, HAL.
    9. Isaiah Andrews & Maximilian Kasy, 2017. "Identification of and correction for publication bias," Papers 1711.10527, arXiv.org.

    More about this item

    Keywords

    misconduct; replicate; reproducible; ethics; open data; transparency; robustness; replication; fraud; error; code; registry;

    JEL classification:

    • B40 - Schools of Economic Thought and Methodology - - Economic Methodology - - - General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • C80 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp9000. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Mark Fallak). General contact details of provider: http://www.iza.org .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.