IDEAS home Printed from https://ideas.repec.org/a/zbw/espost/200119.html
   My bibliography  Save this article

Replication studies in economics—How many and which papers are chosen for replication, and why?

Author

Listed:
  • Mueller-Langer, Frank
  • Fecher, Benedikt
  • Harhoff, Dietmar
  • Wagner, Gert G.

Abstract

We investigate how often replication studies are published in empirical economics and what types of journal articles are replicated. We find that between 1974 and 2014 0.1% of publications in the top 50 economics journals were replication studies. We consider the results of published formal replication studies (whether they are negating or reinforcing) and their extent: Narrow replication studies are typically devoted to mere replication of prior work, while scientific replication studies provide a broader analysis. We find evidence that higher-impact articles and articles by authors from leading institutions are more likely to be replicated, whereas the replication probability is lower for articles that appeared in top 5 economics journals. Our analysis also suggests that mandatory data disclosure policies may have a positive effect on the incidence of replication.

Suggested Citation

  • Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," EconStor Open Access Articles, ZBW - Leibniz Information Centre for Economics, pages 62-83.
  • Handle: RePEc:zbw:espost:200119
    as

    Download full text from publisher

    File URL: https://www.econstor.eu/bitstream/10419/200119/1/Mueller-Langner_2019_Replication%20Studies%20Economics%20FT.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Romer, Paul M, 1986. "Increasing Returns and Long-run Growth," Journal of Political Economy, University of Chicago Press, vol. 94(5), pages 1002-1037, October.
    2. Turan G. Bali & Nusret Cakici & Xuemin (Sterling) Yan & Zhe Zhang, 2005. "Does Idiosyncratic Risk Really Matter?," Journal of Finance, American Finance Association, vol. 60(2), pages 905-929, April.
    3. Benedikt Fecher & Mathis Fräßdorf & Gert G. Wagner, 2016. "Perceptions and Practices of Replication by Social and Behavioral Scientists: Making Replications a Mandatory Element of Curricula Would Be Useful," Discussion Papers of DIW Berlin 1572, DIW Berlin, German Institute for Economic Research.
    4. Lee, Robert H., 2008. "Thinking rigorously about future costs in cost effectiveness analysis," Journal of Health Economics, Elsevier, vol. 27(6), pages 1650-1651, December.
    5. Thomas Herndon & Michael Ash & Robert Pollin, 2014. "Does high public debt consistently stifle economic growth? A critique of Reinhart and Rogoff," Cambridge Journal of Economics, Oxford University Press, vol. 38(2), pages 257-279.
    6. repec:aea:aecrev:v:107:y:2017:i:5:p:60-64 is not listed on IDEAS
    7. Christopher T. Taylor & Nicholas M. Kreisle & Paul R. Zimmerman, 2010. "Vertical Relationships and Competition in Retail Gasoline Markets: Empirical Evidence from Contract Changes in Southern California: Comment," American Economic Review, American Economic Association, vol. 100(3), pages 1269-1276, June.
    8. repec:aea:aecrev:v:107:y:2017:i:5:p:52-55 is not listed on IDEAS
    9. Jacobson, Louis S & LaLonde, Robert J & Sullivan, Daniel G, 1993. "Earnings Losses of Displaced Workers," American Economic Review, American Economic Association, vol. 83(4), pages 685-709, September.
    10. Kenneth A. Couch & Dana W. Placzek, 2010. "Earnings Losses of Displaced Workers Revisited," American Economic Review, American Economic Association, vol. 100(1), pages 572-589, March.
    11. Fehder, Daniel C. & Murray, Fiona & Stern, Scott, 2014. "Intellectual property rights and the evolution of scientific journals as knowledge platforms," International Journal of Industrial Organization, Elsevier, vol. 36(C), pages 83-94.
    12. Ted Joyce, 2004. "Did Legalized Abortion Lower Crime?," Journal of Human Resources, University of Wisconsin Press, vol. 39(1).
    13. Louis S. Jacobson & Robert J. LaLonde & Daniel G. Sullivan, 1993. "Long-term earnings losses of high-seniority displaced workers," Economic Perspectives, Federal Reserve Bank of Chicago, issue Nov, pages 2-20.
    14. Mueller-Langer, Frank & Andreoli-Versbach, Patrick, 2018. "Open access to research data: Strategic delay and the ambiguous welfare effects of mandatory data disclosure," Information Economics and Policy, Elsevier, vol. 42(C), pages 20-34.
    15. Roland G. Fryer Jr. & Paul S. Heaton & Steven D. Levitt & Kevin M. Murphy, 2013. "Measuring Crack Cocaine And Its Impact," Economic Inquiry, Western Economic Association International, vol. 51(3), pages 1651-1681, July.
    16. Fiona Murray & Philippe Aghion & Mathias Dewatripont & Julian Kolev & Scott Stern, 2016. "Of Mice and Academics: Examining the Effect of Openness on Innovation," American Economic Journal: Economic Policy, American Economic Association, vol. 8(1), pages 212-252, February.
    17. Carmen M. Reinhart & Kenneth S. Rogoff, 2010. "Growth in a Time of Debt," American Economic Review, American Economic Association, vol. 100(2), pages 573-578, May.
    18. Carmen M. Reinhart & Vincent R. Reinhart & Kenneth S. Rogoff, 2012. "Public Debt Overhangs: Advanced-Economy Episodes since 1800," Journal of Economic Perspectives, American Economic Association, vol. 26(3), pages 69-86, Summer.
    19. repec:aea:aecrev:v:107:y:2017:i:5:p:41-45 is not listed on IDEAS
    20. John J. Donohue, III & Steven D. Levitt, 2004. "Further Evidence that Legalized Abortion Lowered Crime: A Reply to Joyce," Journal of Human Resources, University of Wisconsin Press, vol. 39(1).
    21. B.D. McCullough & Kerry Anne McGeary & Teresa D. Harrison, 2008. "Do economics journal archives promote replicable research?," Canadian Journal of Economics, Canadian Economics Association, vol. 41(4), pages 1406-1420, November.
    22. repec:aea:aecrev:v:107:y:2017:i:5:p:37-40 is not listed on IDEAS
    23. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    24. John J. Donohue III & Steven D. Levitt, 2001. "The Impact of Legalized Abortion on Crime," The Quarterly Journal of Economics, Oxford University Press, vol. 116(2), pages 379-420.
    25. Jason Abrevaya & Laura Puzzello, 2012. "Taxes, Cigarette Consumption, and Smoking Intensity: Comment," American Economic Review, American Economic Association, vol. 102(4), pages 1751-1763, June.
    26. Catherine Bobtcheff & Jérôme Bolte & Thomas Mariotti, 2017. "Researcher’s Dilemma," Review of Economic Studies, Oxford University Press, vol. 84(3), pages 969-1014.
    27. Stephan, Paula E., 2010. "The Economics of Science," Handbook of the Economics of Innovation, Elsevier.
    28. Dyl, Edward A & Maberly, Edwin D, 1986. " The Weekly Pattern in Stock Index Futures: A Further Note," Journal of Finance, American Finance Association, vol. 41(5), pages 1149-1152, December.
    29. Rock, Steve & Sedo, Stanley & Willenborg, Michael, 2000. "Analyst following and count-data econometrics," Journal of Accounting and Economics, Elsevier, vol. 30(3), pages 351-373, December.
    30. Nicola Lacetera & Lorenzo Zirulia, 2011. "The Economics of Scientific Misconduct," Journal of Law, Economics, and Organization, Oxford University Press, vol. 27(3), pages 568-603.
    31. Christoph Engel, 2011. "Dictator games: a meta study," Experimental Economics, Springer;Economic Science Association, vol. 14(4), pages 583-610, November.
    32. Anderson, Lisa R & Holt, Charles A, 1997. "Information Cascades in the Laboratory," American Economic Review, American Economic Association, vol. 87(5), pages 847-862, December.
    33. Jens Iversen & Ulf S?derstr?m, 2014. "The Dynamic Behavior of the Real Exchange Rate in Sticky Price Models: Comment," American Economic Review, American Economic Association, vol. 104(3), pages 1072-1089, March.
    34. Bhushan, Ravi, 1989. "Firm characteristics and analyst following," Journal of Accounting and Economics, Elsevier, vol. 11(2-3), pages 255-274, July.
    35. repec:aea:aecrev:v:107:y:2017:i:5:p:27-31 is not listed on IDEAS
    36. Adam Wagstaff & Eddy Van Doorslaer, 1994. "Measuring inequalities in health in the presence of multiple‐category morbidity indicators," Health Economics, John Wiley & Sons, Ltd., vol. 3(4), pages 281-291, July.
    37. Sorenson, Olav & Fleming, Lee, 2004. "Science and the diffusion of knowledge," Research Policy, Elsevier, vol. 33(10), pages 1615-1634, December.
    38. Daniel S. Hamermesh, 2017. "Replication in Labor Economics: Evidence from Data, and What It Suggests," American Economic Review, American Economic Association, vol. 107(5), pages 37-40, May.
    39. Maren Duvendack & Richard W. Palmer-Jones & W. Robert Reed, 2015. "Replications in Economics: A Progress Report," Econ Journal Watch, Econ Journal Watch, vol. 12(2), pages 164–191-1, May.
    40. repec:ags:stataj:208012 is not listed on IDEAS
    41. Andreoli-Versbach, Patrick & Mueller-Langer, Frank, 2014. "Open access to data: An ideal professed but not practised," Research Policy, Elsevier, vol. 43(9), pages 1621-1633.
    42. Haeussler, Carolin, 2011. "Information-sharing in academia and the industry: A comparative study," Research Policy, Elsevier, vol. 40(1), pages 105-122, February.
    43. Amit Goyal & Pedro Santa-Clara, 2003. "Idiosyncratic Risk Matters!," Journal of Finance, American Finance Association, vol. 58(3), pages 975-1008, June.
    44. Murray, Fiona & Stern, Scott, 2007. "Do formal intellectual property rights hinder the free flow of scientific knowledge?: An empirical test of the anti-commons hypothesis," Journal of Economic Behavior & Organization, Elsevier, vol. 63(4), pages 648-687, August.
    45. Gerdtham, U. -G. & Johannesson, M. & Lundberg, L. & Isacson, D., 1999. "A note on validating Wagstaff and van Doorslaer's health measure in the analysis of inequalities in health," Journal of Health Economics, Elsevier, vol. 18(1), pages 117-124, January.
    46. Camerer, Colin & Dreber, Anna & Forsell, Eskil & Ho, Teck-Hua & Huber, Jurgen & Johannesson, Magnus & Kirchler, Michael & Almenberg, Johan & Altmejd, Adam & Chan, Taizan & Heikensten, Emma & Holzmeist, 2016. "Evaluating replicability of laboratory experiments in Economics," MPRA Paper 75461, University Library of Munich, Germany.
    47. Fiona E. Murray & Scott Stern, 2007. "Do Formal Intellectual Property Rights Hinder the Free Flow of Scientific Knowledge?: An Empirical Test of the Anti-Commons Hypothesis," NBER Chapters,in: Academic Science and Entrepreneurship: Dual Engines of Growth National Bureau of Economic Research, Inc.
    48. Justine S. Hastings, 2004. "Vertical Relationships and Competition in Retail Gasoline Markets: Empirical Evidence from Contract Changes in Southern California," American Economic Review, American Economic Association, vol. 94(1), pages 317-328, March.
    49. Cornell, Bradford, 1985. " The Weekly Pattern in Stock Returns: Cash versus Futures: A Note," Journal of Finance, American Finance Association, vol. 40(2), pages 583-588, June.
    50. Richard Anderson & William Greene & B. D. McCullough & H. D. Vinod, 2008. "The role of data/code archives in the future of economic research," Journal of Economic Methodology, Taylor & Francis Journals, vol. 15(1), pages 99-119.
    51. Lee, Robert H., 2008. "Future costs in cost effectiveness analysis," Journal of Health Economics, Elsevier, vol. 27(4), pages 809-818, July.
    52. Dewald, William G & Thursby, Jerry G & Anderson, Richard G, 1986. "Replication in Empirical Economics: The Journal of Money, Credit and Banking Project," American Economic Review, American Economic Association, vol. 76(4), pages 587-603, September.
    53. Loasby, Brian J., 2002. "The evolution of knowledge: beyond the biological model," Research Policy, Elsevier, vol. 31(8-9), pages 1227-1239, December.
    54. repec:aea:aecrev:v:107:y:2017:i:5:p:56-59 is not listed on IDEAS
    55. Kiri, Bralind & Lacetera, Nicola & Zirulia, Lorenzo, 2018. "Above a swamp: A theory of high-quality scientific production," Research Policy, Elsevier, vol. 47(5), pages 827-839.
    56. Lutz Bornmann & Rüdiger Mutz, 2015. "Growth rates of modern science: A bibliometric analysis based on the number of publications and cited references," Journal of the Association for Information Science & Technology, Association for Information Science & Technology, vol. 66(11), pages 2215-2222, November.
    57. Dosi, Giovanni, 1988. "Sources, Procedures, and Microeconomic Effects of Innovation," Journal of Economic Literature, American Economic Association, vol. 26(3), pages 1120-1171, September.
    58. Christian Salas & James P. Raftery, 2001. "Econometric issues in testing the age neutrality of health care expenditure," Health Economics, John Wiley & Sons, Ltd., vol. 10(7), pages 669-671.
    59. Gans, Joshua S. & Murray, Fiona E. & Stern, Scott, 2017. "Contracting over the disclosure of scientific knowledge: Intellectual property and academic publication," Research Policy, Elsevier, vol. 46(4), pages 820-835.
    60. Joshua Angrist & Pierre Azoulay & Glenn Ellison & Ryan Hill & Susan Feng Lu, 2017. "Inside Job or Deep Impact? Using Extramural Citations to Assess Economic Scholarship," NBER Working Papers 23698, National Bureau of Economic Research, Inc.
    61. David Card & Stefano DellaVigna, 2013. "Nine Facts about Top Journals in Economics," Journal of Economic Literature, American Economic Association, vol. 51(1), pages 144-161, March.
    62. Partha, Dasgupta & David, Paul A., 1994. "Toward a new economics of science," Research Policy, Elsevier, vol. 23(5), pages 487-521, September.
    63. Andrew Bell & Ron Johnston & Kelvyn Jones, 2015. "Stylised fact or situated messiness? The diverse effects of increasing debt on national economic growth," Journal of Economic Geography, Oxford University Press, vol. 15(2), pages 449-472.
    64. McCullough, B. D. & McGeary, Kerry Anne & Harrison, Teresa D., 2006. "Lessons from the JMCB Archive," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 38(4), pages 1093-1107, June.
    65. Müller-Langer, Frank & Andreoli-Versbach, Patrick, 2018. "Open access to research data: Strategic delay and the ambiguous welfare effects of mandatory data disclosure," Munich Reprints in Economics 62831, University of Munich, Department of Economics.
    66. Furman, Jeffrey L. & Jensen, Kyle & Murray, Fiona, 2012. "Governing knowledge in the scientific community: Exploring the role of retractions in biomedicine," Research Policy, Elsevier, vol. 41(2), pages 276-290.
    67. Jérôme Adda & Francesca Cornaglia, 2006. "Taxes, Cigarette Consumption, and Smoking Intensity," American Economic Review, American Economic Association, vol. 96(4), pages 1013-1028, September.
    68. Lucas C. Coffman & Muriel Niederle, 2015. "Pre-analysis Plans Have Limited Upside, Especially Where Replications Are Feasible," Journal of Economic Perspectives, American Economic Association, vol. 29(3), pages 81-98, Summer.
    69. Vlaeminck, Sven, 2013. "Data Management in Scholarly Journals and Possible Roles for Libraries - Some Insights from EDaWaX," EconStor Open Access Articles, ZBW - Leibniz Information Centre for Economics, pages 49-79.
    70. Ted Joyce, 2009. "A Simple Test of Abortion and Crime," The Review of Economics and Statistics, MIT Press, vol. 91(1), pages 112-123, February.
    71. Christopher L. Foote & Christopher F. Goetz, 2008. "The Impact of Legalized Abortion on Crime: Comment," The Quarterly Journal of Economics, Oxford University Press, vol. 123(1), pages 407-423.
    72. Art Fraas & Randall Lutter, 2012. "Efficient Pollution Regulation: Getting the Prices Right: Comment," American Economic Review, American Economic Association, vol. 102(1), pages 602-607, February.
    73. repec:aea:aecrev:v:107:y:2017:i:5:p:32-36 is not listed on IDEAS
    74. Richard Williams, 2012. "Using the margins command to estimate and interpret adjusted predictions and marginal effects," Stata Journal, StataCorp LP, vol. 12(2), pages 308-331, June.
    75. Daniel S. Hamermesh, 2007. "Viewpoint: Replication in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 40(3), pages 715-733, August.
    76. Hunter, John E, 2001. " The Desperate Need for Replications," Journal of Consumer Research, Oxford University Press, vol. 28(1), pages 149-158, June.
    77. repec:eme:rexezz:s0193-230620150000018008 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    More about this item

    Keywords

    Replication; Economics of science; Science policy; Economic methodology;

    JEL classification:

    • A1 - General Economics and Teaching - - General Economics
    • B4 - Schools of Economic Thought and Methodology - - Economic Methodology
    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C13 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Estimation: General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:espost:200119. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (ZBW - Leibniz Information Centre for Economics). General contact details of provider: http://edirc.repec.org/data/zbwkide.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.