IDEAS home Printed from https://ideas.repec.org/a/ejw/journl/v12y2015i2p164-191.html
   My bibliography  Save this article

Replications in Economics: A Progress Report

Author

Listed:
  • Maren Duvendack
  • Richard W. Palmer-Jones
  • W. Robert Reed

Abstract

We report on various aspects of replication research in economics. Our report includes (i) a brief history of data sharing and replication; (ii) the results of a survey administered to the editors of all 333 economics journals listed in Web of Science; (iii) an analysis of 162 replication studies that have been published in peer-reviewed economics journals from 1977–2014; (iv) a discussion of the future of replication research in economics; and (v) observations on how replications can be better integrated into research efforts to address problems associated with publication bias and other Type I error phenomena. This paper is part of an ongoing project which includes the website replicationnetwork.com, which provides additional, regularly updated information on replications in economics.

Suggested Citation

  • Maren Duvendack & Richard W. Palmer-Jones & W. Robert Reed, 2015. "Replications in Economics: A Progress Report," Econ Journal Watch, Econ Journal Watch, vol. 12(2), pages 164–191-1, May.
    • Handle: RePEc:ejw:journl:v:12:y:2015:i:2:p:164-191
    as

    Download full text from publisher

    File URL: https://econjwatch.org/File+download/866/DuvendackEtAlMay2015.pdf?mimetype=pdf
    Download Restriction: no
    File URL: https://econjwatch.org/983
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Charles R. Link & Edward C. Ratledge, 1975. "Social Returns to Quantity and Quality of Education: A Further Statement," Journal of Human Resources, University of Wisconsin Press, vol. 10(1), pages 78-89.
    2. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    3. Stan J. Liebowitz, 2014. "Willful Blindness: The Inefficient Reward Structure In Academic Research," Economic Inquiry, Western Economic Association International, vol. 52(4), pages 1267-1283, October.
    4. Vegard Iversen & Richard Palmer-Jones, 2008. "Literacy Sharing, Assortative Mating, or What? Labour Market Advantages and Proximate Illiteracy Revisited," Journal of Development Studies, Taylor & Francis Journals, vol. 44(6), pages 797-838.
    5. T. D. Stanley & Stephen B. Jarrell, 2005. "Meta‐Regression Analysis: A Quantitative Method of Literature Surveys," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 299-308, July.
    6. Card, David & Krueger, Alan B, 1995. "Time-Series Minimum-Wage Studies: A Meta-analysis," American Economic Review, American Economic Association, vol. 85(2), pages 238-243, May.
    7. David Y. Albouy, 2012. "The Colonial Origins of Comparative Development: An Empirical Investigation: Comment," American Economic Review, American Economic Association, vol. 102(6), pages 3059-3076, October.
    8. Lowell E. Gallaway & Zachary Dyckman, 1970. "The Full Employment-Unemployment Rate: 1953-1980," Journal of Human Resources, University of Wisconsin Press, vol. 5(4), pages 487-510.
    9. Laura Camfield & Richard Palmer-Jones, 2013. "Three 'Rs' of Econometrics: Repetition, Reproduction and Replication," Journal of Development Studies, Taylor & Francis Journals, vol. 49(12), pages 1607-1614, December.
    10. Robert Jensen & Emily Oster, 2009. "The Power of TV: Cable Television and Women's Status in India," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 124(3), pages 1057-1094.
    11. Leonard E. Burman & W. Robert Reed & James Alm, 2011. "A Call for Replication Studies," Public Finance Review, , vol. 39(1), pages 190-190, January.
    12. John S. Akin & Thomas J. Kniesner, 1976. "Proxies for Observations on Individuals Sampled from a Population," Journal of Human Resources, University of Wisconsin Press, vol. 11(3), pages 411-413.
    13. David W. Findlay & John M. Santos, 2012. "Supplement to "Race, Ethnicity, and Baseball Card Prices: A Replication, Correction, and Extension of Hewitt, Muñoz, Oliver, and Regoli"," Econ Journal Watch, Econ Journal Watch, vol. 9(2), pages 1-18, May.
    14. Orley Ashenfelter & Colm Harmon & Hessel Oosterbeek, 1999. "A Review of Estimates of the Schooling/Earnings Relationship, with Tests for Publication Bias," Working Papers 804, Princeton University, Department of Economics, Industrial Relations Section..
    15. Peter Gordon & Lanlan Wang, 2004. "Does Economic Performance Correlate with Big Government?," Econ Journal Watch, Econ Journal Watch, vol. 1(2), pages 192-221, August.
    16. Charles R. Link & Edward C. Ratledge, 1976. "Proxies for Observations on Individuals Sampled from a Population: Reply," Journal of Human Resources, University of Wisconsin Press, vol. 11(3), pages 413-419.
    17. Justin McCrary, 2002. "Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime: Comment," American Economic Review, American Economic Association, vol. 92(4), pages 1236-1243, September.
    18. Feige, Edgar L, 1975. "The Consequences of Journal Editorial Policies and a Suggestion for Revision," Journal of Political Economy, University of Chicago Press, vol. 83(6), pages 1291-1295, December.
    19. Maren Duvendack & Richard Palmer-Jones, 2013. "Replication of quantitative work in development studies: Experiences and suggestions," Progress in Development Studies, , vol. 13(4), pages 307-322, October.
    20. repec:fth:prinin:425 is not listed on IDEAS
    21. Chris Doucouliagos, 2005. "Publication Bias in the Economic Freedom and Economic Growth Literature," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 367-387, July.
    22. Siskind, Frederic B, 1977. "Minimum Wage Legislation in the United States: Comment," Economic Inquiry, Western Economic Association International, vol. 15(1), pages 135-138, January.
    23. Marshall S. Smith, 1968. "Equality of Educational Opportunity: Comments on Bowles and Levin," Journal of Human Resources, University of Wisconsin Press, vol. 3(3), pages 384-389.
    24. Richard Anderson & William Greene & B. D. McCullough & H. D. Vinod, 2008. "The role of data/code archives in the future of economic research," Journal of Economic Methodology, Taylor & Francis Journals, vol. 15(1), pages 99-119.
    25. Dewald, William G & Thursby, Jerry G & Anderson, Richard G, 1986. "Replication in Empirical Economics: The Journal of Money, Credit and Banking Project," American Economic Review, American Economic Association, vol. 76(4), pages 587-603, September.
    26. Jan H. Höffler, 2014. "Teaching Replication in Quantitative Empirical Economics," Replication Working Papers 2/2014, Institut für Statistik und Ökonometrie, Wirtschaftswissenschaftliche Fakultät, Georg-August-Universität Göttingen, Replication project.
    27. B. D. McCullough & H. D. Vinod, 2003. "Verifying the Solution from a Nonlinear Solver: A Case Study," American Economic Review, American Economic Association, vol. 93(3), pages 873-892, June.
    28. Baltagi, Badi H & Khanti-Akom, Sophon, 1990. "On Efficient Estimation with Panel Data: An Empirical Comparison of Instrumental Variables Estimators," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 5(4), pages 401-406, Oct.-Dec..
    29. Daron Acemoglu & Simon Johnson & James A. Robinson, 2012. "The Colonial Origins of Comparative Development: An Empirical Investigation: Reply," American Economic Review, American Economic Association, vol. 102(6), pages 3077-3110, October.
    30. Richard Jong-A-Pin & Jakob De Haan, 2008. "Growth Accelerations and Regime Changes: A Correction," Econ Journal Watch, Econ Journal Watch, vol. 5(1), pages 51-58, January.
    31. John Gibson & David L. Anderson & John Tressler, 2014. "Which Journal Rankings Best Explain Academic Salaries? Evidence From The University Of California," Economic Inquiry, Western Economic Association International, vol. 52(4), pages 1322-1340, October.
    32. Donald R. Winkler, 1975. "Educational Achievement and School Peer Group Composition," Journal of Human Resources, University of Wisconsin Press, vol. 10(2), pages 189-204.
    33. B.D. McCullough, 2009. "Open Access Economics Journals and the Market for Reproducible Economic Research," Economic Analysis and Policy, Elsevier, vol. 39(1), pages 117-126, March.
    34. McCullough, B. D. & McGeary, Kerry Anne & Harrison, Teresa D., 2006. "Lessons from the JMCB Archive," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 38(4), pages 1093-1107, June.
    35. Ashenfelter, Orley & Harmon, Colm & Oosterbeek, Hessel, 1999. "A review of estimates of the schooling/earnings relationship, with tests for publication bias," Labour Economics, Elsevier, vol. 6(4), pages 453-470, November.
    36. Caroline M. Hoxby, 2007. "Does Competition Among Public Schools Benefit Students and Taxpayers? Reply," American Economic Review, American Economic Association, vol. 97(5), pages 2038-2055, December.
    37. Andreas Ortmann & John Fitzgerald & Carl Boeing, 2000. "Trust, Reciprocity, and Social History: A Re-examination," Experimental Economics, Springer;Economic Science Association, vol. 3(1), pages 81-100, June.
    38. Basu, Kaushik & Narayan, Ambar & Ravallion, Martin, 2001. "Is literacy shared within households? Theory and evidence for Bangladesh," Labour Economics, Elsevier, vol. 8(6), pages 649-665, December.
    39. Lant Pritchett, 2002. "It pays to be ignorant: A simple political economy of rigorous program evaluation," Journal of Economic Policy Reform, Taylor & Francis Journals, vol. 5(4), pages 251-269.
    40. Jesse Rothstein, 2007. "Does Competition Among Public Schools Benefit Students and Taxpayers? Comment," American Economic Review, American Economic Association, vol. 97(5), pages 2026-2037, December.
    41. Mayer, Thomas, 1980. "Economics as a Hard Science: Realistic Goal or Wishful Thinking?," Economic Inquiry, Western Economic Association International, vol. 18(2), pages 165-178, April.
    42. John Ioannidis & Chris Doucouliagos, 2013. "What'S To Know About The Credibility Of Empirical Economics?," Journal of Economic Surveys, Wiley Blackwell, vol. 27(5), pages 997-1004, December.
    43. Zacharias Maniadis & Fabio Tufano & John A. List, 2014. "One Swallow Doesn't Make a Summer: New Evidence on Anchoring Effects," American Economic Review, American Economic Association, vol. 104(1), pages 277-290, January.
    44. Mina Bissell, 2013. "Reproducibility: The risks of the replication drive," Nature, Nature, vol. 503(7476), pages 333-334, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Maren Duvendack & Richard Palmer-Jones, 2013. "Replication of quantitative work in development studies: Experiences and suggestions," Progress in Development Studies, , vol. 13(4), pages 307-322, October.
    2. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    3. Michael A. Clemens, 2017. "The Meaning Of Failed Replications: A Review And Proposal," Journal of Economic Surveys, Wiley Blackwell, vol. 31(1), pages 326-342, February.
    4. Christopher Snyder & Ran Zhuo, 2018. "Sniff Tests as a Screen in the Publication Process: Throwing out the Wheat with the Chaff," NBER Working Papers 25058, National Bureau of Economic Research, Inc.
    5. Mueller-Langer, Frank & Andreoli-Versbach, Patrick, 2018. "Open access to research data: Strategic delay and the ambiguous welfare effects of mandatory data disclosure," Information Economics and Policy, Elsevier, vol. 42(C), pages 20-34.
    6. Stanley, T. D. & Doucouliagos, Chris, 2019. "Practical Significance, Meta-Analysis and the Credibility of Economics," IZA Discussion Papers 12458, Institute of Labor Economics (IZA).
    7. Andreoli-Versbach, Patrick & Mueller-Langer, Frank, 2014. "Open access to data: An ideal professed but not practised," Research Policy, Elsevier, vol. 43(9), pages 1621-1633.
    8. Mark J. McCabe & Frank Mueller-Langer, 2019. "Does Data Disclosure Increase Citations? Empirical Evidence from a Natural Experiment in Leading Economics Journals," JRC Working Papers on Digital Economy 2019-02, Joint Research Centre (Seville site).
    9. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    10. Valérie Orozco & Christophe Bontemps & Élise Maigné & Virginie Piguet & Annie Hofstetter & Anne Marie Lacroix & Fabrice Levert & Jean-Marc Rousselle, 2017. "How to make a pie? Reproducible Research for Empirical Economics & Econometrics," Post-Print hal-01939942, HAL.
    11. Alan S. Gerber & Neil Malhotra, 2008. "Publication Bias in Empirical Sociological Research," Sociological Methods & Research, , vol. 37(1), pages 3-30, August.
    12. Hristos Doucouliagos & T. D. Stanley, 2009. "Publication Selection Bias in Minimum‐Wage Research? A Meta‐Regression Analysis," British Journal of Industrial Relations, London School of Economics, vol. 47(2), pages 406-428, June.
    13. Vlaeminck, Sven, 2013. "Data Management in Scholarly Journals and Possible Roles for Libraries - Some Insights from EDaWaX," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 23(1), pages 49-79.
    14. Christian Zimmermann, 2015. "On the Need for a Replication Journal," Working Papers 2015-16, Federal Reserve Bank of St. Louis.
    15. Andrew C. Chang & Phillip Li, 2015. "Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say \"Usually Not\"," Finance and Economics Discussion Series 2015-83, Board of Governors of the Federal Reserve System (U.S.).
    16. Valérie Orozco & Christophe Bontemps & Elise Maigné & Virginie Piguet & Annie Hofstetter & Anne Lacroix & Fabrice Levert & Jean‐Marc Rousselle, 2020. "How To Make A Pie: Reproducible Research For Empirical Economics And Econometrics," Journal of Economic Surveys, Wiley Blackwell, vol. 34(5), pages 1134-1169, December.
    17. Fidrmuc, Jarko & Lind, Ronja, 2020. "Macroeconomic impact of Basel III: Evidence from a meta-analysis," Journal of Banking & Finance, Elsevier, vol. 112(C).
    18. Sebri, Maamar, 2015. "Use renewables to be cleaner: Meta-analysis of the renewable energy consumption–economic growth nexus," Renewable and Sustainable Energy Reviews, Elsevier, vol. 42(C), pages 657-665.
    19. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    20. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," Research Policy, Elsevier, vol. 48(1), pages 62-83.

    More about this item

    Keywords

    Replication; data sharing; publication bias;
    All these keywords.

    JEL classification:

    • A1 - General Economics and Teaching - - General Economics
    • B4 - Schools of Economic Thought and Methodology - - Economic Methodology

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ejw:journl:v:12:y:2015:i:2:p:164-191. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Jason Briggeman (email available below). General contact details of provider: https://edirc.repec.org/data/edgmuus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.