IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp9000.html
   My bibliography  Save this paper

The Meaning of Failed Replications: A Review and Proposal

Author

Listed:
  • Clemens, Michael A.

    (George Mason University)

Abstract

The welcome rise of replication tests in economics has not been accompanied by a single, clear definition of replication. A discrepant replication, in current usage of the term, can signal anything from an unremarkable disagreement over methods to scientific incompetence or misconduct. This paper proposes an unambiguous definition of replication, one that reflects currently common but unstandardized use. It contrasts this definition with decades of unsuccessful attempts to standardize terminology, and argues that many prominent results described as replication tests – in labor, development, and other fields of economics – should not be described as such. Adopting this definition can improve incentives for researchers, encouraging more and better replication tests.

Suggested Citation

  • Clemens, Michael A., 2015. "The Meaning of Failed Replications: A Review and Proposal," IZA Discussion Papers 9000, Institute of Labor Economics (IZA).
    • Handle: RePEc:iza:izadps:dp9000
    as

    Download full text from publisher

    File URL: https://docs.iza.org/dp9000.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Michael Ross, 2006. "Is Democracy Good for the Poor?," American Journal of Political Science, John Wiley & Sons, vol. 50(4), pages 860-874, October.
    2. Thomas Herndon & Michael Ash & Robert Pollin, 2014. "Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff," Cambridge Journal of Economics, Cambridge Political Economy Society, vol. 38(2), pages 257-279.
    3. John R. Lott & John Whitley, 2007. "Abortion And Crime: Unwanted Children And Out‐Of‐Wedlock Births," Economic Inquiry, Western Economic Association International, vol. 45(2), pages 304-324, April.
    4. Edward D. Mansfield & Helen V. Milner & B. Peter Rosendorff, 2015. "Free to Trade: Democracies, Autocracies, and International Trade," World Scientific Book Chapters, in: Edward D Mansfield (ed.), THE POLITICAL ECONOMY OF INTERNATIONAL TRADE, chapter 7, pages 127-143, World Scientific Publishing Co. Pte. Ltd..
    5. David Y. Albouy, 2012. "The Colonial Origins of Comparative Development: An Empirical Investigation: Comment," American Economic Review, American Economic Association, vol. 102(6), pages 3059-3076, October.
    6. Ted Joyce, 2004. "Did Legalized Abortion Lower Crime?," Journal of Human Resources, University of Wisconsin Press, vol. 39(1).
    7. Carmen M. Reinhart & Kenneth S. Rogoff, 2010. "Growth in a Time of Debt," American Economic Review, American Economic Association, vol. 100(2), pages 573-578, May.
    8. Summers, Lawrence H, 1991. " The Scientific Illusion in Empirical Macroeconomics," Scandinavian Journal of Economics, Wiley Blackwell, vol. 93(2), pages 129-148.
    9. David Dollar & Craig Burnside, 2000. "Aid, Policies, and Growth," American Economic Review, American Economic Association, vol. 90(4), pages 847-868, September.
    10. Johnson, Simon & Larson, William & Papageorgiou, Chris & Subramanian, Arvind, 2013. "Is newer better? Penn World Table Revisions and their impact on growth estimates," Journal of Monetary Economics, Elsevier, vol. 60(2), pages 255-274.
    11. Evanschitzky, Heiner & Baumgarth, Carsten & Hubbard, Raymond & Armstrong, J. Scott, 2007. "Replication research's disturbing trend," Journal of Business Research, Elsevier, vol. 60(4), pages 411-415, April.
    12. H. M. Collins, 1991. "The Meaning of Replication and the Science of Economics," History of Political Economy, Duke University Press, vol. 23(1), pages 123-142, Spring.
    13. Daron Acemoglu & Joshua D. Angrist, 2001. "Consequences of Employment Protection? The Case of the Americans with Disabilities Act," Journal of Political Economy, University of Chicago Press, vol. 109(5), pages 915-957, October.
    14. Justin McCrary, 2002. "Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime: Comment," American Economic Review, American Economic Association, vol. 92(4), pages 1236-1243, September.
    15. Benjamin F. Jones & Benjamin A. Olken, 2005. "Do Leaders Matter? National Leadership and Growth Since World War II," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 120(3), pages 835-864.
    16. Munnell, Alicia H. & Geoffrey M. B. Tootell & Lynn E. Browne & James McEneaney, 1996. "Mortgage Lending in Boston: Interpreting HMDA Data," American Economic Review, American Economic Association, vol. 86(1), pages 25-53, March.
    17. John Lott & John Whitley, "undated". "Abortion and Crime: Unwanted Children and Out-of-Wedlock Births," Yale Law School John M. Olin Center for Studies in Law, Economics, and Public Policy Working Paper Series yale_lepp-1018, Yale Law School John M. Olin Center for Studies in Law, Economics, and Public Policy.
    18. Balẳ Égert, 2015. "The 90% public debt threshold: the rise and fall of a stylized fact," Applied Economics, Taylor & Francis Journals, vol. 47(34-35), pages 3756-3770, July.
    19. Daron Acemoglu & Simon Johnson & James A. Robinson, 2001. "The Colonial Origins of Comparative Development: An Empirical Investigation," American Economic Review, American Economic Association, vol. 91(5), pages 1369-1401, December.
    20. Graham A. Davis, 2013. "Replicating Sachs and Warner's Working Papers on the Resource Curse," Journal of Development Studies, Taylor & Francis Journals, vol. 49(12), pages 1615-1630, December.
    21. Basu, Kaushik & Kanbur, Ravi (ed.), 2008. "Arguments for a Better World: Essays in Honor of Amartya Sen: Volume I: Ethics, Welfare, and Measurement," OUP Catalogue, Oxford University Press, number 9780199239115.
    22. Abhijit Banerjee & Lakshmi Iyer, 2005. "History, Institutions, and Economic Performance: The Legacy of Colonial Land Tenure Systems in India," American Economic Review, American Economic Association, vol. 95(4), pages 1190-1213, September.
    23. Dewald, William G & Thursby, Jerry G & Anderson, Richard G, 1986. "Replication in Empirical Economics: The Journal of Money, Credit and Banking Project," American Economic Review, American Economic Association, vol. 76(4), pages 587-603, September.
    24. A. W. Phillips, 1958. "The Relation Between Unemployment and the Rate of Change of Money Wage Rates in the United Kingdom, 1861–1957," Economica, London School of Economics and Political Science, vol. 25(100), pages 283-299, November.
    25. Nancy J. Wulwick, 1996. "Two Econometric Replications: The Historic Phillips and Lipsey-Phillips Curves," History of Political Economy, Duke University Press, vol. 28(3), pages 391-439, Fall.
    26. Stefan Dercon & Daniel O. Gilligan & John Hoddinott & Tassew Woldehanna, 2009. "The Impact of Agricultural Extension and Roads on Poverty and Consumption Growth in Fifteen Ethiopian Villages," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 91(4), pages 1007-1021.
    27. Hatice Ozer Balli & Bent Sørensen, 2013. "Interaction effects in econometrics," Empirical Economics, Springer, vol. 45(1), pages 583-603, August.
    28. Edward Miguel & Michael Kremer, 2004. "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities," Econometrica, Econometric Society, vol. 72(1), pages 159-217, January.
    29. Boone, Peter, 1996. "Politics and the effectiveness of foreign aid," European Economic Review, Elsevier, vol. 40(2), pages 289-329, February.
    30. Levitt, Steven D, 1997. "Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime," American Economic Review, American Economic Association, vol. 87(3), pages 270-290, June.
    31. William Easterly & Ross Levine & David Roodman, 2004. "Aid, Policies, and Growth: Comment," American Economic Review, American Economic Association, vol. 94(3), pages 774-780, June.
    32. Boyce, James K & Ravallion, Martin, 1991. "A Dynamic Econometric Model of Agricultural Wage Determination in Bangladesh," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 53(4), pages 361-376, November.
    33. Antonio Ciccone, 2011. "Economic Shocks and Civil Conflict: A Comment," American Economic Journal: Applied Economics, American Economic Association, vol. 3(4), pages 215-227, October.
    34. repec:cup:apsrev:v:96:y:2002:i:01:p:159-165_00 is not listed on IDEAS
    35. Harrison, Glenn W, 1998. "Mortgage Lending in Boston: A Reconsideration of the Evidence," Economic Inquiry, Western Economic Association International, vol. 36(1), pages 29-38, January.
    36. Michael A. Clemens & Steven Radelet & Rikhil R. Bhavnani & Samuel Bazzi, 2012. "Counting Chickens when they Hatch: Timing and the Effects of Aid on Growth," Economic Journal, Royal Economic Society, vol. 122(561), pages 590-617, June.
    37. Caroline M. Hoxby, 2000. "Does Competition among Public Schools Benefit Students and Taxpayers?," American Economic Review, American Economic Association, vol. 90(5), pages 1209-1238, December.
    38. Arulampalam, Wiji & Hartog, Joop & MaCurdy, Tom & Theeuwes, Jules, 1997. "Replication and re-analysis," Labour Economics, Elsevier, vol. 4(2), pages 99-105, June.
    39. Katherine Casey & Rachel Glennerster & Edward Miguel, 2012. "Reshaping Institutions: Evidence on Aid Impacts Using a Preanalysis Plan," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 127(4), pages 1755-1812.
    40. Nancy Cartwright, 1991. "Replicability, Reproducibility, and Robustness: Comments on Harry Collins," History of Political Economy, Duke University Press, vol. 23(1), pages 143-155, Spring.
    41. Easley, Richard W. & Madden, Charles S., 2000. "Replications and Extensions in Marketing and Management Research," Journal of Business Research, Elsevier, vol. 48(1), pages 1-3, April.
    42. Patrick Barwise, 1995. "Good Empirical Generalizations," Marketing Science, INFORMS, vol. 14(3_supplem), pages 29-35.
    43. B.D. McCullough & Kerry Anne McGeary & Teresa D. Harrison, 2008. "Do economics journal archives promote replicable research?," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 41(4), pages 1406-1420, November.
    44. Basu, Kaushik & Kanbur, Ravi (ed.), 2008. "Arguments for a Better World: Essays in Honor of Amartya Sen: Volume II: Society, Institutions, and Development," OUP Catalogue, Oxford University Press, number 9780199239979.
    45. Hamermesh, Daniel S., 1997. "Some thoughts on replications and reviews," Labour Economics, Elsevier, vol. 4(2), pages 107-109, June.
    46. Ming-Jen Lin & Ming-Ching Luoh, 2008. "Can Hepatitis B Mothers Account for the Number of Missing Women? Evidence from Three Million Newborns in Taiwan," American Economic Review, American Economic Association, vol. 98(5), pages 2259-2273, December.
    47. A. G. Sleeman, 2011. "Retrospectives: The Phillips Curve: A Rushed Job?," Journal of Economic Perspectives, American Economic Association, vol. 25(1), pages 223-238, Winter.
    48. Feldstein, Martin S, 1974. "Social Security, Induced Retirement, and Aggregate Capital Accumulation," Journal of Political Economy, University of Chicago Press, vol. 82(5), pages 905-926, Sept./Oct.
    49. Hubbard, Raymond & Vetter, Daniel E., 1996. "An empirical comparison of published replication research in accounting, economics, finance, management, and marketing," Journal of Business Research, Elsevier, vol. 35(2), pages 153-164, February.
    50. Richard Anderson & William Greene & B. D. McCullough & H. D. Vinod, 2008. "The role of data/code archives in the future of economic research," Journal of Economic Methodology, Taylor & Francis Journals, vol. 15(1), pages 99-119.
    51. Leimer, Dean R & Lesnoy, Selig D, 1982. "Social Security and Private Saving: New Time-Series Evidence," Journal of Political Economy, University of Chicago Press, vol. 90(3), pages 606-629, June.
    52. Bruce Chapman & Yvonne Dunlop & Matthew Gray & Amy Liu & Deborah Mitchell, 2001. "The Impact of Children on the Lifetime Earnings of Australian Women: Evidence from the 1990s," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 34(4), pages 373-389, December.
    53. Furman, Jeffrey L. & Jensen, Kyle & Murray, Fiona, 2012. "Governing knowledge in the scientific community: Exploring the role of retractions in biomedicine," Research Policy, Elsevier, vol. 41(2), pages 276-290.
    54. Samuel Bazzi & Michael A. Clemens, 2013. "Blunt Instruments: Avoiding Common Pitfalls in Identifying the Causes of Economic Growth," American Economic Journal: Macroeconomics, American Economic Association, vol. 5(2), pages 152-186, April.
    55. Caroline M. Hoxby, 2007. "Does Competition Among Public Schools Benefit Students and Taxpayers? Reply," American Economic Review, American Economic Association, vol. 97(5), pages 2038-2055, December.
    56. Annette N. Brown & Drew B. Cameron & Benjamin D. K. Wood, 2014. "Quality evidence for policymaking: I'll believe it when I see the replication," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(3), pages 215-235, September.
    57. Jesse Rothstein, 2007. "Does Competition Among Public Schools Benefit Students and Taxpayers? Comment," American Economic Review, American Economic Association, vol. 97(5), pages 2026-2037, December.
    58. Martha J. Bailey, 2006. "More Power to the Pill: The Impact of Contraceptive Freedom on Women's Life Cycle Labor Supply," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 121(1), pages 289-320.
    59. John Ioannidis & Chris Doucouliagos, 2013. "What'S To Know About The Credibility Of Empirical Economics?," Journal of Economic Surveys, Wiley Blackwell, vol. 27(5), pages 997-1004, December.
    60. Edward Miguel & Shanker Satyanath & Ernest Sergenti, 2004. "Economic Shocks and Civil Conflict: An Instrumental Variables Approach," Journal of Political Economy, University of Chicago Press, vol. 112(4), pages 725-753, August.
    61. Farley, John U & Lehmann, Donald R & Ryan, Michael J, 1981. "Generalizing from "Imperfect" Replication," The Journal of Business, University of Chicago Press, vol. 54(4), pages 597-610, October.
    62. Hunter, John E, 2001. "The Desperate Need for Replications," Journal of Consumer Research, Journal of Consumer Research Inc., vol. 28(1), pages 149-158, June.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Maren Duvendack & Richard Palmer-Jones, 2013. "Replication of quantitative work in development studies: Experiences and suggestions," Progress in Development Studies, , vol. 13(4), pages 307-322, October.
    2. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    3. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," Research Policy, Elsevier, vol. 48(1), pages 62-83.
    4. Daniel S. Hamermesh, 2007. "Viewpoint: Replication in economics," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 40(3), pages 715-733, August.
    5. Andrew C. Chang & Phillip Li, 2015. "Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say \"Usually Not\"," Finance and Economics Discussion Series 2015-83, Board of Governors of the Federal Reserve System (U.S.).
    6. B.D. McCullough & Kerry Anne McGeary & Teresa D. Harrison, 2008. "Do economics journal archives promote replicable research?," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 41(4), pages 1406-1420, November.
    7. Hamermesh, Daniel S., 2007. "Replication in Economics," IZA Discussion Papers 2760, Institute of Labor Economics (IZA).
    8. Maren Duvendack & Richard W. Palmer-Jones & W. Robert Reed, 2015. "Replications in Economics: A Progress Report," Econ Journal Watch, Econ Journal Watch, vol. 12(2), pages 164–191-1, May.
    9. Mark J. McCabe & Frank Mueller-Langer, 2019. "Does Data Disclosure Increase Citations? Empirical Evidence from a Natural Experiment in Leading Economics Journals," JRC Working Papers on Digital Economy 2019-02, Joint Research Centre.
    10. Annette N. Brown & Drew B. Cameron & Benjamin D. K. Wood, 2014. "Quality evidence for policymaking: I'll believe it when I see the replication," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(3), pages 215-235, September.
    11. Sebastian Galiani & Stephen Knack & Lixin Colin Xu & Ben Zou, 2017. "The effect of aid on growth: evidence from a Quasi-experiment," Journal of Economic Growth, Springer, vol. 22(1), pages 1-33, March.
    12. Valérie Orozco & Christophe Bontemps & Elise Maigné & Virginie Piguet & Annie Hofstetter & Anne Lacroix & Fabrice Levert & Jean‐Marc Rousselle, 2020. "How To Make A Pie: Reproducible Research For Empirical Economics And Econometrics," Journal of Economic Surveys, Wiley Blackwell, vol. 34(5), pages 1134-1169, December.
    13. Calderon,Cesar & Nguyen,Ha Minh, 2015. "Do capital inflows boost growth in developing countries ? evidence from Sub-Saharan Africa," Policy Research Working Paper Series 7298, The World Bank.
    14. Juergen Bitzer & Erkan Goeren, 2018. "Foreign Aid and Subnational Development: A Grid Cell Analysis," Working Papers V-407-18, University of Oldenburg, Department of Economics, revised Mar 2018.
    15. Timothy Besley & Torsten Persson, 2011. "Pillars of Prosperity: The Political Economics of Development Clusters," Economics Books, Princeton University Press, edition 1, number 9624.
    16. Temple, Jonathan R.W., 2010. "Aid and Conditionality," Handbook of Development Economics, in: Dani Rodrik & Mark Rosenzweig (ed.), Handbook of Development Economics, edition 1, volume 5, chapter 0, pages 4415-4523, Elsevier.
    17. Andreoli-Versbach, Patrick & Mueller-Langer, Frank, 2014. "Open access to data: An ideal professed but not practised," Research Policy, Elsevier, vol. 43(9), pages 1621-1633.
    18. Angus Deaton, 2009. "Instruments of development: Randomization in the tropics, and the search for the elusive keys to economic development," Working Papers 1128, Princeton University, Woodrow Wilson School of Public and International Affairs, Center for Health and Wellbeing..
    19. Drazen, Allan & Dreber, Anna & Ozbay, Erkut Y. & Snowberg, Erik, 2021. "Journal-based replication of experiments: An application to “Being Chosen to Lead”," Journal of Public Economics, Elsevier, vol. 202(C).
    20. Christophe Hurlin & Christophe Pérignon, 2020. "Reproducibility Certification in Economics Research," Working Papers hal-02896404, HAL.

    More about this item

    Keywords

    replication; robustness; transparency; open data; ethics; reproducible; replicate; misconduct; fraud; error; code; registry;
    All these keywords.

    JEL classification:

    • B40 - Schools of Economic Thought and Methodology - - Economic Methodology - - - General
    • C18 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Methodolical Issues: General
    • C80 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp9000. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Holger Hinte (email available below). General contact details of provider: https://edirc.repec.org/data/izaaade.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.