IDEAS home Printed from https://ideas.repec.org/a/bla/jecsur/v31y2017i1p326-342.html
   My bibliography  Save this article

The Meaning Of Failed Replications: A Review And Proposal

Author

Listed:
  • Michael A. Clemens

Abstract

The welcome rise of replication tests in economics has not been accompanied by a single, clear definition of replication. A discrepant replication, in current usage of the term, can signal anything from an unremarkable disagreement over methods to scientific incompetence or misconduct. This paper proposes an unambiguous definition of replication, one that reflects currently common but unstandardized use. It contrasts this definition with decades of unsuccessful attempts to standardize terminology, and argues that many prominent results described as replication tests – in labor, development, and other fields of economics – should not be described as such. Adopting this definition can improve incentives for researchers, encouraging more and better replication tests.
(This abstract was borrowed from another version of this item.)

Suggested Citation

  • Michael A. Clemens, 2017. "The Meaning Of Failed Replications: A Review And Proposal," Journal of Economic Surveys, Wiley Blackwell, vol. 31(1), pages 326-342, February.
  • Handle: RePEc:bla:jecsur:v:31:y:2017:i:1:p:326-342
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1111/joes.2017.31.issue-1
    Download Restriction: Access to full text is restricted to subscribers.
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Michael Ross, 2006. "Is Democracy Good for the Poor?," American Journal of Political Science, John Wiley & Sons, vol. 50(4), pages 860-874, October.
    2. Harrison, Glenn W, 1998. "Mortgage Lending in Boston: A Reconsideration of the Evidence," Economic Inquiry, Western Economic Association International, vol. 36(1), pages 29-38, January.
    3. Hubbard, Raymond & Vetter, Daniel E., 1996. "An empirical comparison of published replication research in accounting, economics, finance, management, and marketing," Journal of Business Research, Elsevier, vol. 35(2), pages 153-164, February.
    4. Thomas Herndon & Michael Ash & Robert Pollin, 2014. "Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff," Cambridge Journal of Economics, Oxford University Press, vol. 38(2), pages 257-279.
    5. Richard Anderson & William Greene & B. D. McCullough & H. D. Vinod, 2008. "The role of data/code archives in the future of economic research," Journal of Economic Methodology, Taylor & Francis Journals, vol. 15(1), pages 99-119.
    6. Daron Acemoglu & Simon Johnson & James A. Robinson, 2001. "The Colonial Origins of Comparative Development: An Empirical Investigation," American Economic Review, American Economic Association, vol. 91(5), pages 1369-1401, December.
    7. John R. Lott & John Whitley, 2007. "Abortion And Crime: Unwanted Children And Out‐Of‐Wedlock Births," Economic Inquiry, Western Economic Association International, vol. 45(2), pages 304-324, April.
    8. Leimer, Dean R & Lesnoy, Selig D, 1982. "Social Security and Private Saving: New Time-Series Evidence," Journal of Political Economy, University of Chicago Press, vol. 90(3), pages 606-629, June.
    9. Ted Joyce, 2004. "Did Legalized Abortion Lower Crime?," Journal of Human Resources, University of Wisconsin Press, vol. 39(1).
    10. Nancy Cartwright, 1991. "Replicability, Reproducibility, and Robustness: Comments on Harry Collins," History of Political Economy, Duke University Press, vol. 23(1), pages 143-155, Spring.
    11. Bruce Chapman & Yvonne Dunlop & Matthew Gray & Amy Liu & Deborah Mitchell, 2001. "The Impact of Children on the Lifetime Earnings of Australian Women: Evidence from the 1990s," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 34(4), pages 373-389, December.
    12. Levitt, Steven D, 1997. "Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime," American Economic Review, American Economic Association, vol. 87(3), pages 270-290, June.
    13. Hatice Ozer Balli & Bent Sørensen, 2013. "Interaction effects in econometrics," Empirical Economics, Springer, vol. 45(1), pages 583-603, August.
    14. Edward Miguel & Michael Kremer, 2004. "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities," Econometrica, Econometric Society, vol. 72(1), pages 159-217, January.
    15. Boone, Peter, 1996. "Politics and the effectiveness of foreign aid," European Economic Review, Elsevier, vol. 40(2), pages 289-329, February.
    16. Munnell, Alicia H. & Geoffrey M. B. Tootell & Lynn E. Browne & James McEneaney, 1996. "Mortgage Lending in Boston: Interpreting HMDA Data," American Economic Review, American Economic Association, vol. 86(1), pages 25-53, March.
    17. B.D. McCullough & Kerry Anne McGeary & Teresa D. Harrison, 2008. "Do economics journal archives promote replicable research?," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 41(4), pages 1406-1420, November.
    18. Summers, Lawrence H, 1991. " The Scientific Illusion in Empirical Macroeconomics," Scandinavian Journal of Economics, Wiley Blackwell, vol. 93(2), pages 129-148.
    19. Furman, Jeffrey L. & Jensen, Kyle & Murray, Fiona, 2012. "Governing knowledge in the scientific community: Exploring the role of retractions in biomedicine," Research Policy, Elsevier, vol. 41(2), pages 276-290.
    20. William Easterly & Ross Levine & David Roodman, 2004. "Aid, Policies, and Growth: Comment," American Economic Review, American Economic Association, vol. 94(3), pages 774-780, June.
    21. Hamermesh, Daniel S., 1997. "Some thoughts on replications and reviews," Labour Economics, Elsevier, vol. 4(2), pages 107-109, June.
    22. Johnson, Simon & Larson, William & Papageorgiou, Chris & Subramanian, Arvind, 2013. "Is newer better? Penn World Table Revisions and their impact on growth estimates," Journal of Monetary Economics, Elsevier, vol. 60(2), pages 255-274.
    23. Caroline M. Hoxby, 2007. "Does Competition Among Public Schools Benefit Students and Taxpayers? Reply," American Economic Review, American Economic Association, vol. 97(5), pages 2038-2055, December.
    24. Evanschitzky, Heiner & Baumgarth, Carsten & Hubbard, Raymond & Armstrong, J. Scott, 2007. "Replication research's disturbing trend," Journal of Business Research, Elsevier, vol. 60(4), pages 411-415, April.
    25. Annette N. Brown & Drew B. Cameron & Benjamin D. K. Wood, 2014. "Quality evidence for policymaking: I'll believe it when I see the replication," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(3), pages 215-235, September.
    26. Ming-Jen Lin & Ming-Ching Luoh, 2008. "Can Hepatitis B Mothers Account for the Number of Missing Women? Evidence from Three Million Newborns in Taiwan," American Economic Review, American Economic Association, vol. 98(5), pages 2259-2273, December.
    27. Jesse Rothstein, 2007. "Does Competition Among Public Schools Benefit Students and Taxpayers? Comment," American Economic Review, American Economic Association, vol. 97(5), pages 2026-2037, December.
    28. Martha J. Bailey, 2006. "More Power to the Pill: The Impact of Contraceptive Freedom on Women's Life Cycle Labor Supply," The Quarterly Journal of Economics, Oxford University Press, vol. 121(1), pages 289-320.
    29. Justin McCrary, 2002. "Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime: Comment," American Economic Review, American Economic Association, vol. 92(4), pages 1236-1243, September.
    30. Feldstein, Martin S, 1974. "Social Security, Induced Retirement, and Aggregate Capital Accumulation," Journal of Political Economy, University of Chicago Press, vol. 82(5), pages 905-926, Sept./Oct.
    31. Farley, John U & Lehmann, Donald R & Ryan, Michael J, 1981. "Generalizing from "Imperfect" Replication," The Journal of Business, University of Chicago Press, vol. 54(4), pages 597-610, October.
    32. Mansfield, Edward D. & Milner, Helen V. & Rosendorff, B. Peter, 2000. "Free to Trade: Democracies, Autocracies, and International Trade," American Political Science Review, Cambridge University Press, vol. 94(2), pages 305-321, June.
    33. Graham A. Davis, 2013. "Replicating Sachs and Warner's Working Papers on the Resource Curse," Journal of Development Studies, Taylor & Francis Journals, vol. 49(12), pages 1615-1630, December.
    34. Hunter, John E, 2001. "The Desperate Need for Replications," Journal of Consumer Research, Oxford University Press, vol. 28(1), pages 149-158, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," Research Policy, Elsevier, vol. 48(1), pages 62-83.
    2. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    3. Christensen, Garret & Miguel, Edward & Sturdy, Jennifer, 2017. "Transparency, Reproducibility, and the Credibility of Economics Research," MetaArXiv 9a3rw, Center for Open Science.
    4. Annette N. Brown & Drew B. Cameron & Benjamin D. K. Wood, 2014. "Quality evidence for policymaking: I'll believe it when I see the replication," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(3), pages 215-235, September.
    5. Mark J. McCabe & Frank Mueller-Langer, 2019. "Does Data Disclosure Increase Citations? Empirical Evidence from a Natural Experiment in Leading Economics Journals," JRC Working Papers on Digital Economy 2019-02, Joint Research Centre (Seville site).
    6. Daniel S. Hamermesh, 2007. "Viewpoint: Replication in economics," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 40(3), pages 715-733, August.
    7. Andreoli-Versbach, Patrick & Mueller-Langer, Frank, 2014. "Open access to data: An ideal professed but not practised," Research Policy, Elsevier, vol. 43(9), pages 1621-1633.
    8. Maren Duvendack & Richard W. Palmer-Jones & W. Robert Reed, 2015. "Replications in Economics: A Progress Report," Econ Journal Watch, Econ Journal Watch, vol. 12(2), pages 164–191-1, May.
    9. B.D. McCullough & Kerry Anne McGeary & Teresa D. Harrison, 2008. "Do economics journal archives promote replicable research?," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 41(4), pages 1406-1420, November.
    10. Hamermesh, Daniel S., 2007. "Replication in Economics," IZA Discussion Papers 2760, Institute of Labor Economics (IZA).
    11. Edinaldo Tebaldi & Erik Alda, 2017. "Quality of Institutions and Violence Incidence: a Cross-Country Analysis," Atlantic Economic Journal, Springer;International Atlantic Economic Society, vol. 45(3), pages 365-384, September.
    12. Andrew C. Chang & Phillip Li, 2015. "Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say \"Usually Not\"," Finance and Economics Discussion Series 2015-83, Board of Governors of the Federal Reserve System (U.S.).
    13. Alejandro Gaviria & Carlos Medina & Jorge Tamayo, 2010. "Assessing the Link between Adolescent Fertility and Urban Crime," BORRADORES DE ECONOMIA 006860, BANCO DE LA REPÚBLICA.
    14. Clark, Andrew E. & Milcent, Carine, 2011. "Public employment and political pressure: The case of French hospitals," Journal of Health Economics, Elsevier, vol. 30(5), pages 1103-1112.
    15. Mueller-Langer, Frank & Andreoli-Versbach, Patrick, 2018. "Open access to research data: Strategic delay and the ambiguous welfare effects of mandatory data disclosure," Information Economics and Policy, Elsevier, vol. 42(C), pages 20-34.
    16. Calderon,Cesar & Nguyen,Ha Minh, 2015. "Do capital inflows boost growth in developing countries ? evidence from Sub-Saharan Africa," Policy Research Working Paper Series 7298, The World Bank.
    17. Temple, Jonathan R.W., 2010. "Aid and Conditionality," Handbook of Development Economics, in: Dani Rodrik & Mark Rosenzweig (ed.), Handbook of Development Economics, edition 1, volume 5, chapter 0, pages 4415-4523, Elsevier.
    18. Angeles, Luis & Neanidis, Kyriakos C., 2009. "Aid effectiveness: the role of the local elite," Journal of Development Economics, Elsevier, vol. 90(1), pages 120-134, September.
    19. Robert Chernomas & Ian Hudson, 2013. "Steven Levitt on Abortion and Crime: Old Economics in New Bottles," American Journal of Economics and Sociology, Wiley Blackwell, vol. 72(3), pages 675-700, July.
    20. Evanschitzky, Heiner & Armstrong, J. Scott, 2010. "Replications of forecasting research," International Journal of Forecasting, Elsevier, vol. 26(1), pages 4-8, January.

    More about this item

    JEL classification:

    • B40 - Schools of Economic Thought and Methodology - - Economic Methodology - - - General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • C80 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jecsur:v:31:y:2017:i:1:p:326-342. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: http://www.blackwellpublishing.com/journal.asp?ref=0950-0804 .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: http://www.blackwellpublishing.com/journal.asp?ref=0950-0804 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.