IDEAS home Printed from https://ideas.repec.org/
MyIDEAS: Log in (now much improved!) to save this paper

The Meaning of Failed Replications: A Review and Proposal

Listed author(s):
  • Clemens, Michael A.

    ()

    (Center for Global Development)

The welcome rise of replication tests in economics has not been accompanied by a single, clear definition of replication. A discrepant replication, in current usage of the term, can signal anything from an unremarkable disagreement over methods to scientific incompetence or misconduct. This paper proposes an unambiguous definition of replication, one that reflects currently common but unstandardized use. It contrasts this definition with decades of unsuccessful attempts to standardize terminology, and argues that many prominent results described as replication tests – in labor, development, and other fields of economics – should not be described as such. Adopting this definition can improve incentives for researchers, encouraging more and better replication tests.

If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.

File URL: http://ftp.iza.org/dp9000.pdf
Download Restriction: no

Paper provided by Institute for the Study of Labor (IZA) in its series IZA Discussion Papers with number 9000.

as
in new window

Length: 27 pages
Date of creation: Apr 2015
Publication status: published in: Journal of Economic Surveys, 2017, 31 (1): 326-342
Handle: RePEc:iza:izadps:dp9000
Contact details of provider: Postal:
IZA, P.O. Box 7240, D-53072 Bonn, Germany

Phone: +49 228 3894 223
Fax: +49 228 3894 180
Web page: http://www.iza.org

Order Information: Postal: IZA, Margard Ody, P.O. Box 7240, D-53072 Bonn, Germany
Email:


References listed on IDEAS
Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:

as
in new window


  1. Harrison, Glenn W, 1998. "Mortgage Lending in Boston: A Reconsideration of the Evidence," Economic Inquiry, Western Economic Association International, vol. 36(1), pages 29-38, January.
  2. Hubbard, Raymond & Vetter, Daniel E., 1996. "An empirical comparison of published replication research in accounting, economics, finance, management, and marketing," Journal of Business Research, Elsevier, vol. 35(2), pages 153-164, February.
  3. Thomas Herndon & Michael Ash & Robert Pollin, 2014. "Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff," Cambridge Journal of Economics, Oxford University Press, vol. 38(2), pages 257-279.
  4. Richard Anderson & William Greene & B. D. McCullough & H. D. Vinod, 2008. "The role of data/code archives in the future of economic research," Journal of Economic Methodology, Taylor & Francis Journals, vol. 15(1), pages 99-119.
  5. Daron Acemoglu & Simon Johnson & James A. Robinson, 2001. "The Colonial Origins of Comparative Development: An Empirical Investigation," American Economic Review, American Economic Association, vol. 91(5), pages 1369-1401, December.
  6. John R. Lott & John Whitley, 2007. "Abortion And Crime: Unwanted Children And Out-Of-Wedlock Births," Economic Inquiry, Western Economic Association International, vol. 45(2), pages 304-324, 04.
  7. Leimer, Dean R & Lesnoy, Selig D, 1982. "Social Security and Private Saving: New Time-Series Evidence," Journal of Political Economy, University of Chicago Press, vol. 90(3), pages 606-629, June.
  8. Ted Joyce, 2004. "Did Legalized Abortion Lower Crime?," Journal of Human Resources, University of Wisconsin Press, vol. 39(1).
  9. Nancy Cartwright, 1991. "Replicability, Reproducibility, and Robustness: Comments on Harry Collins," History of Political Economy, Duke University Press, vol. 23(1), pages 143-155, Spring.
  10. Bruce Chapman & Yvonne Dunlop & Matthew Gray & Amy Liu & Deborah Mitchell, 2001. "The Impact of Children on the Lifetime Earnings of Australian Women: Evidence from the 1990s," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 34(4), pages 373-389.
  11. Levitt, Steven D, 1997. "Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime," American Economic Review, American Economic Association, vol. 87(3), pages 270-290, June.
  12. Hatice Balli & Bent Sørensen, 2013. "Interaction effects in econometrics," Empirical Economics, Springer, vol. 45(1), pages 583-603, August.
  13. Edward Miguel & Michael Kremer, 2004. "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities," Econometrica, Econometric Society, vol. 72(1), pages 159-217, 01.
  14. Boone, Peter, 1996. "Politics and the effectiveness of foreign aid," European Economic Review, Elsevier, vol. 40(2), pages 289-329, February.
  15. Munnell, Alicia H. & Geoffrey M. B. Tootell & Lynn E. Browne & James McEneaney, 1996. "Mortgage Lending in Boston: Interpreting HMDA Data," American Economic Review, American Economic Association, vol. 86(1), pages 25-53, March.
  16. B.D. McCullough & Kerry Anne McGeary & Teresa D. Harrison, 2008. "Do economics journal archives promote replicable research?," Canadian Journal of Economics, Canadian Economics Association, vol. 41(4), pages 1406-1420, November.
  17. Summers, Lawrence H, 1991. " The Scientific Illusion in Empirical Macroeconomics," Scandinavian Journal of Economics, Wiley Blackwell, vol. 93(2), pages 129-148.
  18. Furman, Jeffrey L. & Jensen, Kyle & Murray, Fiona, 2012. "Governing knowledge in the scientific community: Exploring the role of retractions in biomedicine," Research Policy, Elsevier, vol. 41(2), pages 276-290.
  19. William Easterly & Ross Levine & David Roodman, 2004. "Aid, Policies, and Growth: Comment," American Economic Review, American Economic Association, vol. 94(3), pages 774-780, June.
  20. Hamermesh, Daniel S., 1997. "Some thoughts on replications and reviews," Labour Economics, Elsevier, vol. 4(2), pages 107-109, June.
  21. Johnson, Simon & Larson, William & Papageorgiou, Chris & Subramanian, Arvind, 2013. "Is newer better? Penn World Table Revisions and their impact on growth estimates," Journal of Monetary Economics, Elsevier, vol. 60(2), pages 255-274.
  22. Caroline M. Hoxby, 2007. "Does Competition Among Public Schools Benefit Students and Taxpayers? Reply," American Economic Review, American Economic Association, vol. 97(5), pages 2038-2055, December.
  23. Evanschitzky, Heiner & Baumgarth, Carsten & Hubbard, Raymond & Armstrong, J. Scott, 2007. "Replication research's disturbing trend," Journal of Business Research, Elsevier, vol. 60(4), pages 411-415, April.
  24. Annette N. Brown & Drew B. Cameron & Benjamin D. K. Wood, 2014. "Quality evidence for policymaking: I'll believe it when I see the replication," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(3), pages 215-235, September.
  25. Ming-Jen Lin & Ming-Ching Luoh, 2008. "Can Hepatitis B Mothers Account for the Number of Missing Women? Evidence from Three Million Newborns in Taiwan," American Economic Review, American Economic Association, vol. 98(5), pages 2259-2273, December.
  26. Jesse Rothstein, 2007. "Does Competition Among Public Schools Benefit Students and Taxpayers? Comment," American Economic Review, American Economic Association, vol. 97(5), pages 2026-2037, December.
  27. Martha J. Bailey, 2006. "More Power to the Pill: The Impact of Contraceptive Freedom on Women's Life Cycle Labor Supply," The Quarterly Journal of Economics, Oxford University Press, vol. 121(1), pages 289-320.
  28. Justin McCrary, 2002. "Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime: Comment," American Economic Review, American Economic Association, vol. 92(4), pages 1236-1243, September.
  29. Feldstein, Martin S, 1974. "Social Security, Induced Retirement, and Aggregate Capital Accumulation," Journal of Political Economy, University of Chicago Press, vol. 82(5), pages 905-926, Sept./Oct.
  30. Farley, John U & Lehmann, Donald R & Ryan, Michael J, 1981. "Generalizing from "Imperfect" Replication," The Journal of Business, University of Chicago Press, vol. 54(4), pages 597-610, October.
  31. Graham A. Davis, 2013. "Replicating Sachs and Warner's Working Papers on the Resource Curse," Journal of Development Studies, Taylor & Francis Journals, vol. 49(12), pages 1615-1630, December.
  32. Hunter, John E, 2001. " The Desperate Need for Replications," Journal of Consumer Research, Oxford University Press, vol. 28(1), pages 149-158, June.
Full references (including those not matched with items on IDEAS)

This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.

When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp9000. See general information about how to correct material in RePEc.

For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Mark Fallak)

If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

If references are entirely missing, you can add them using this form.

If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

Please note that corrections may take a couple of weeks to filter through the various RePEc services.

This information is provided to you by IDEAS at the Research Division of the Federal Reserve Bank of St. Louis using RePEc data.