IDEAS home Printed from https://ideas.repec.org/a/ejw/journl/v12y2015i2p164-191.html
   My bibliography  Save this article

Replications in Economics: A Progress Report

Author

Listed:
  • Maren Duvendack
  • Richard W. Palmer-Jones
  • W. Robert Reed

Abstract

We report on various aspects of replication research in economics. Our report includes (i) a brief history of data sharing and replication; (ii) the results of a survey administered to the editors of all 333 economics journals listed in Web of Science; (iii) an analysis of 162 replication studies that have been published in peer-reviewed economics journals from 1977–2014; (iv) a discussion of the future of replication research in economics; and (v) observations on how replications can be better integrated into research efforts to address problems associated with publication bias and other Type I error phenomena. This paper is part of an ongoing project which includes the website replicationnetwork.com, which provides additional, regularly updated information on replications in economics.

Suggested Citation

  • Maren Duvendack & Richard W. Palmer-Jones & W. Robert Reed, 2015. "Replications in Economics: A Progress Report," Econ Journal Watch, Econ Journal Watch, vol. 12(2), pages 164–191-1, May.
  • Handle: RePEc:ejw:journl:v:12:y:2015:i:2:p:164-191
    as

    Download full text from publisher

    File URL: https://econjwatch.org/File+download/866/DuvendackEtAlMay2015.pdf?mimetype=pdf
    Download Restriction: no

    File URL: https://econjwatch.org/983
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Charles R. Link & Edward C. Ratledge, 1975. "Social Returns to Quantity and Quality of Education: A Further Statement," Journal of Human Resources, University of Wisconsin Press, vol. 10(1), pages 78-89.
    2. Stan J. Liebowitz, 2014. "Willful Blindness: The Inefficient Reward Structure In Academic Research," Economic Inquiry, Western Economic Association International, vol. 52(4), pages 1267-1283, October.
    3. Vegard Iversen & Richard Palmer-Jones, 2008. "Literacy Sharing, Assortative Mating, or What? Labour Market Advantages and Proximate Illiteracy Revisited," Journal of Development Studies, Taylor & Francis Journals, vol. 44(6), pages 797-838.
    4. T. D. Stanley & Stephen B. Jarrell, 2005. "Meta-Regression Analysis: A Quantitative Method of Literature Surveys," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 299-308, July.
    5. Card, David & Krueger, Alan B, 1995. "Time-Series Minimum-Wage Studies: A Meta-analysis," American Economic Review, American Economic Association, vol. 85(2), pages 238-243, May.
    6. David Y. Albouy, 2012. "The Colonial Origins of Comparative Development: An Empirical Investigation: Comment," American Economic Review, American Economic Association, vol. 102(6), pages 3059-3076, October.
    7. Lowell E. Gallaway & Zachary Dyckman, 1970. "The Full Employment-Unemployment Rate: 1953-1980," Journal of Human Resources, University of Wisconsin Press, vol. 5(4), pages 487-510.
    8. Laura Camfield & Richard Palmer-Jones, 2013. "Three 'Rs' of Econometrics: Repetition, Reproduction and Replication," Journal of Development Studies, Taylor & Francis Journals, vol. 49(12), pages 1607-1614, December.
    9. Robert Jensen & Emily Oster, 2009. "The Power of TV: Cable Television and Women's Status in India," The Quarterly Journal of Economics, Oxford University Press, vol. 124(3), pages 1057-1094.
    10. Leonard E. Burman & W. Robert Reed & James ALm, 2011. "A Call for Replication Studies," Public Finance Review, , vol. 39(1), pages 190-190, January.
    11. John S. Akin & Thomas J. Kniesner, 1976. "Proxies for Observations on Individuals Sampled from a Population," Journal of Human Resources, University of Wisconsin Press, vol. 11(3), pages 411-413.
    12. David W. Findlay & John M. Santos, 2012. "Race, Ethnicity, and Baseball Card Prices: A Replication, Correction, and Extension of Hewitt, Muñoz, Oliver, and Regoli," Econ Journal Watch, Econ Journal Watch, vol. 9(2), pages 122-140, May.
    13. Orley Ashenfelter & Colm Harmon & Hessel Oosterbeek, 1999. "A Review of Estimates of the Schooling/Earnings Relationship, with Tests for Publication Bias," Working Papers 804, Princeton University, Department of Economics, Industrial Relations Section..
    14. Peter Gordon & Lanlan Wang, 2004. "Does Economic Performance Correlate with Big Government?," Econ Journal Watch, Econ Journal Watch, vol. 1(2), pages 192-221, August.
    15. Charles R. Link & Edward C. Ratledge, 1976. "Proxies for Observations on Individuals Sampled from a Population: Reply," Journal of Human Resources, University of Wisconsin Press, vol. 11(3), pages 413-419.
    16. Justin McCrary, 2002. "Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime: Comment," American Economic Review, American Economic Association, vol. 92(4), pages 1236-1243, September.
    17. Feige, Edgar L, 1975. "The Consequences of Journal Editorial Policies and a Suggestion for Revision," Journal of Political Economy, University of Chicago Press, vol. 83(6), pages 1291-1295, December.
    18. Maren Duvendack & Richard Palmer-Jones, 2013. "Replication of quantitative work in development studies: Experiences and suggestions," Progress in Development Studies, , vol. 13(4), pages 307-322, October.
    19. repec:fth:prinin:425 is not listed on IDEAS
    20. Chris Doucouliagos, 2005. "Publication Bias in the Economic Freedom and Economic Growth Literature," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 367-387, July.
    21. Siskind, Frederic B, 1977. "Minimum Wage Legislation in the United States: Comment," Economic Inquiry, Western Economic Association International, vol. 15(1), pages 135-138, January.
    22. Marshall S. Smith, 1968. "Equality of Educational Opportunity: Comments on Bowles and Levin," Journal of Human Resources, University of Wisconsin Press, vol. 3(3), pages 384-389.
    23. Richard Anderson & William Greene & B. D. McCullough & H. D. Vinod, 2008. "The role of data/code archives in the future of economic research," Journal of Economic Methodology, Taylor & Francis Journals, vol. 15(1), pages 99-119.
    24. Jan H. Höffler, 2014. "Teaching Replication in Quantitative Empirical Economics," Replication Working Papers 2/2014, Institut für Statistik und Ökonometrie, Wirtschaftswissenschaftliche Fakultät, Georg-August-Universität Göttingen, Replication project.
    25. B. D. McCullough & H. D. Vinod, 2003. "Verifying the Solution from a Nonlinear Solver: A Case Study," American Economic Review, American Economic Association, vol. 93(3), pages 873-892, June.
    26. Baltagi, Badi H & Khanti-Akom, Sophon, 1990. "On Efficient Estimation with Panel Data: An Empirical Comparison of Instrumental Variables Estimators," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 5(4), pages 401-406, Oct.-Dec..
    27. Daron Acemoglu & Simon Johnson & James A. Robinson, 2012. "The Colonial Origins of Comparative Development: An Empirical Investigation: Reply," American Economic Review, American Economic Association, vol. 102(6), pages 3077-3110, October.
    28. Richard Jong-A-Pin & Jakob De Haan, 2008. "Growth Accelerations and Regime Changes: A Correction," Econ Journal Watch, Econ Journal Watch, vol. 5(1), pages 51-58, January.
    29. John Gibson & David L. Anderson & John Tressler, 2014. "Which Journal Rankings Best Explain Academic Salaries? Evidence From The University Of California," Economic Inquiry, Western Economic Association International, vol. 52(4), pages 1322-1340, October.
    30. Donald R. Winkler, 1975. "Educational Achievement and School Peer Group Composition," Journal of Human Resources, University of Wisconsin Press, vol. 10(2), pages 189-204.
    31. McCullough, B. D. & McGeary, Kerry Anne & Harrison, Teresa D., 2006. "Lessons from the JMCB Archive," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 38(4), pages 1093-1107, June.
    32. Ashenfelter, Orley & Harmon, Colm & Oosterbeek, Hessel, 1999. "A review of estimates of the schooling/earnings relationship, with tests for publication bias," Labour Economics, Elsevier, vol. 6(4), pages 453-470, November.
    33. Caroline M. Hoxby, 2007. "Does Competition Among Public Schools Benefit Students and Taxpayers? Reply," American Economic Review, American Economic Association, vol. 97(5), pages 2038-2055, December.
    34. Andreas Ortmann & John Fitzgerald & Carl Boeing, 2000. "Trust, Reciprocity, and Social History: A Re-examination," Experimental Economics, Springer;Economic Science Association, vol. 3(1), pages 81-100, June.
    35. Basu, Kaushik & Narayan, Ambar & Ravallion, Martin, 2001. "Is literacy shared within households? Theory and evidence for Bangladesh," Labour Economics, Elsevier, vol. 8(6), pages 649-665, December.
    36. Lant Pritchett, 2002. "It pays to be ignorant: A simple political economy of rigorous program evaluation," Journal of Economic Policy Reform, Taylor & Francis Journals, vol. 5(4), pages 251-269.
    37. Jesse Rothstein, 2007. "Does Competition Among Public Schools Benefit Students and Taxpayers? Comment," American Economic Review, American Economic Association, vol. 97(5), pages 2026-2037, December.
    38. Mayer, Thomas, 1980. "Economics as a Hard Science: Realistic Goal or Wishful Thinking?," Economic Inquiry, Western Economic Association International, vol. 18(2), pages 165-178, April.
    39. John Ioannidis & Chris Doucouliagos, 2013. "What'S To Know About The Credibility Of Empirical Economics?," Journal of Economic Surveys, Wiley Blackwell, vol. 27(5), pages 997-1004, December.
    40. Zacharias Maniadis & Fabio Tufano & John A. List, 2014. "One Swallow Doesn't Make a Summer: New Evidence on Anchoring Effects," American Economic Review, American Economic Association, vol. 104(1), pages 277-290, January.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. repec:eee:respol:v:48:y:2019:i:1:p:62-83 is not listed on IDEAS
    2. Maren Duvendack & Richard Palmer-Jones & W. Robert Reed, 2017. "What Is Meant by "Replication" and Why Does It Encounter Resistance in Economics?," American Economic Review, American Economic Association, vol. 107(5), pages 46-51, May.
    3. Zimmermann, Christian, 2015. "On the Need for a Replication Journal," Working Papers 2015-16, Federal Reserve Bank of St. Louis.
    4. repec:lrk:eeaart:36_1_22 is not listed on IDEAS
    5. Nicolas Vallois & Dorian Jullien, 2017. "Replication in experimental economics: A historical and quantitative approach focused on public good game experiments," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) halshs-01651080, HAL.
    6. repec:zbw:espost:200119 is not listed on IDEAS
    7. repec:iza:izawol:journl:y:2017:n:413 is not listed on IDEAS
    8. Frank Mueller-Langer & Benedikt Fecher & Dietmar Harhoff & Gert G. Wagner, 2017. "The Economics of Replication," Discussion Papers of DIW Berlin 1640, DIW Berlin, German Institute for Economic Research.
    9. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," Research Policy, Elsevier, vol. 48(1), pages 62-83.
    10. Fecher, Benedikt & Fräßdorf, Mathis & Wagner, Gert G., 2016. "Perceptions and Practices of Replication by Social and Behavioral Scientists: Making Replications a Mandatory Element of Curricula Would Be Useful," IZA Discussion Papers 9896, Institute of Labor Economics (IZA).
    11. W. Robert Reed, 2017. "Replication in labor economics," IZA World of Labor, Institute of Labor Economics (IZA), pages 413-413, December.
    12. repec:bla:ausecr:v:51:y:2018:i:2:p:286-300 is not listed on IDEAS
    13. repec:zbw:espost:177391 is not listed on IDEAS
    14. Nicolas Vallois & Dorian Jullien, 2017. "Replication in experimental economics: A historical and quantitative approach focused on public good game experiments," Working Papers halshs-01651080, HAL.
    15. Hirschauer Norbert & Grüner Sven & Mußhoff Oliver & Frey Ulrich & Theesfeld Insa & Wagner Peter, 2016. "Die Interpretation des p-Wertes – Grundsätzliche Missverständnisse," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 236(5), pages 557-575, October.
    16. repec:wly:econjl:v:127:y:2017:i:605:p:f209-f235 is not listed on IDEAS
    17. Nicolas Vallois & Dorian Jullien, 2017. "Replication in Experimental Economics: A Historical and Quantitative Approach Focused on Public Good Game Experiments," GREDEG Working Papers 2017-21, Groupe de REcherche en Droit, Economie, Gestion (GREDEG CNRS), University of Nice Sophia Antipolis.
    18. Zacharias Maniadis & Fabio Tufano & John A. List, 2017. "To Replicate or Not To Replicate? Exploring Reproducibility in Economics through the Lens of a Model and a Pilot Study," Economic Journal, Royal Economic Society, vol. 127(605), pages 209-235, October.
    19. repec:eee:poleco:v:54:y:2018:i:c:p:5-15 is not listed on IDEAS
    20. Winker Peter, 2016. "Editorial Announcement," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 236(1), pages 1-2, February.

    More about this item

    Keywords

    Replication; data sharing; publication bias;

    JEL classification:

    • A1 - General Economics and Teaching - - General Economics
    • B4 - Schools of Economic Thought and Methodology - - Economic Methodology

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ejw:journl:v:12:y:2015:i:2:p:164-191. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Jason Briggeman). General contact details of provider: http://edirc.repec.org/data/edgmuus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.