IDEAS home Printed from https://ideas.repec.org/a/ejw/journl/v12y2015i2p164-191.html
   My bibliography  Save this article

Replications in Economics: A Progress Report

Author

Listed:
  • Maren Duvendack
  • Richard W. Palmer-Jones
  • W. Robert Reed

Abstract

We report on various aspects of replication research in economics. Our report includes (i) a brief history of data sharing and replication; (ii) the results of a survey administered to the editors of all 333 economics journals listed in Web of Science; (iii) an analysis of 162 replication studies that have been published in peer-reviewed economics journals from 1977–2014; (iv) a discussion of the future of replication research in economics; and (v) observations on how replications can be better integrated into research efforts to address problems associated with publication bias and other Type I error phenomena. This paper is part of an ongoing project which includes the website replicationnetwork.com, which provides additional, regularly updated information on replications in economics.

Suggested Citation

  • Maren Duvendack & Richard W. Palmer-Jones & W. Robert Reed, 2015. "Replications in Economics: A Progress Report," Econ Journal Watch, Econ Journal Watch, vol. 12(2), pages 164–191-1, May.
  • Handle: RePEc:ejw:journl:v:12:y:2015:i:2:p:164-191
    as

    Download full text from publisher

    File URL: https://econjwatch.org/File+download/866/DuvendackEtAlMay2015.pdf?mimetype=pdf
    Download Restriction: no

    File URL: https://econjwatch.org/983
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. Charles R. Link & Edward C. Ratledge, 1975. "Social Returns to Quantity and Quality of Education: A Further Statement," Journal of Human Resources, University of Wisconsin Press, vol. 10(1), pages 78-89.
    2. Stan J. Liebowitz, 2014. "Willful Blindness: The Inefficient Reward Structure In Academic Research," Economic Inquiry, Western Economic Association International, vol. 52(4), pages 1267-1283, October.
    3. Vegard Iversen & Richard Palmer-Jones, 2008. "Literacy Sharing, Assortative Mating, or What? Labour Market Advantages and Proximate Illiteracy Revisited," Journal of Development Studies, Taylor & Francis Journals, vol. 44(6), pages 797-838.
    4. Card, David & Krueger, Alan B, 1995. "Time-Series Minimum-Wage Studies: A Meta-analysis," American Economic Review, American Economic Association, vol. 85(2), pages 238-243, May.
    5. David Y. Albouy, 2012. "The Colonial Origins of Comparative Development: An Empirical Investigation: Comment," American Economic Review, American Economic Association, vol. 102(6), pages 3059-3076, October.
    6. Lowell E. Gallaway & Zachary Dyckman, 1970. "The Full Employment-Unemployment Rate: 1953-1980," Journal of Human Resources, University of Wisconsin Press, vol. 5(4), pages 487-510.
    7. Laura Camfield & Richard Palmer-Jones, 2013. "Three 'Rs' of Econometrics: Repetition, Reproduction and Replication," Journal of Development Studies, Taylor & Francis Journals, vol. 49(12), pages 1607-1614, December.
    8. Robert Jensen & Emily Oster, 2009. "The Power of TV: Cable Television and Women's Status in India," The Quarterly Journal of Economics, Oxford University Press, vol. 124(3), pages 1057-1094.
    9. Leonard E. Burman & W. Robert Reed & James Alm, 2010. "A Call for Replication Studies," Public Finance Review, , vol. 38(6), pages 787-793, November.
    10. John S. Akin & Thomas J. Kniesner, 1976. "Proxies for Observations on Individuals Sampled from a Population," Journal of Human Resources, University of Wisconsin Press, vol. 11(3), pages 411-413.
    11. David W. Findlay & John M. Santos, 2012. "Race, Ethnicity, and Baseball Card Prices: A Replication, Correction, and Extension of Hewitt, Muñoz, Oliver, and Regoli," Econ Journal Watch, Econ Journal Watch, vol. 9(2), pages 122-140, May.
    12. Orley Ashenfelter & Colm Harmon & Hessel Oosterbeek, 1999. "A Review of Estimates of the Schooling/Earnings Relationship, with Tests for Publication Bias," Working Papers 804, Princeton University, Department of Economics, Industrial Relations Section..
    13. Peter Gordon & Lanlan Wang, 2004. "Does Economic Performance Correlate with Big Government?," Econ Journal Watch, Econ Journal Watch, vol. 1(2), pages 192-221, August.
    14. Charles R. Link & Edward C. Ratledge, 1976. "Proxies for Observations on Individuals Sampled from a Population: Reply," Journal of Human Resources, University of Wisconsin Press, vol. 11(3), pages 413-419.
    15. Justin McCrary, 2002. "Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime: Comment," American Economic Review, American Economic Association, vol. 92(4), pages 1236-1243, September.
    16. Feige, Edgar L, 1975. "The Consequences of Journal Editorial Policies and a Suggestion for Revision," Journal of Political Economy, University of Chicago Press, vol. 83(6), pages 1291-1295, December.
    17. Maren Duvendack & Richard Palmer-Jones, 2013. "Replication of quantitative work in development studies: Experiences and suggestions," Progress in Development Studies, , vol. 13(4), pages 307-322, October.
    18. repec:fth:prinin:425 is not listed on IDEAS
    19. Chris Doucouliagos, 2005. "Publication Bias in the Economic Freedom and Economic Growth Literature," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 367-387, July.
    20. Siskind, Frederic B, 1977. "Minimum Wage Legislation in the United States: Comment," Economic Inquiry, Western Economic Association International, vol. 15(1), pages 135-138, January.
    21. Marshall S. Smith, 1968. "Equality of Educational Opportunity: Comments on Bowles and Levin," Journal of Human Resources, University of Wisconsin Press, vol. 3(3), pages 384-389.
    22. Richard Anderson & William Greene & B. D. McCullough & H. D. Vinod, 2008. "The role of data/code archives in the future of economic research," Journal of Economic Methodology, Taylor & Francis Journals, vol. 15(1), pages 99-119.
    23. Stanley, T D & Jarrell, Stephen B, 1989. "Meta-Regression Analysis: A Quantitative Method of Literature Survey s," Journal of Economic Surveys, Wiley Blackwell, vol. 3(2), pages 161-170.
    24. Jan H. Höffler, 2014. "Teaching Replication in Quantitative Empirical Economics," Replication Working Papers 2/2014, Institut für Statistik und Ökonometrie, Wirtschaftswissenschaftliche Fakultät, Georg-August-Universität Göttingen, Replication project.
    25. B. D. McCullough & H. D. Vinod, 2003. "Verifying the Solution from a Nonlinear Solver: A Case Study," American Economic Review, American Economic Association, vol. 93(3), pages 873-892, June.
    26. Baltagi, Badi H & Khanti-Akom, Sophon, 1990. "On Efficient Estimation with Panel Data: An Empirical Comparison of Instrumental Variables Estimators," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 5(4), pages 401-406, Oct.-Dec..
    27. Daron Acemoglu & Simon Johnson & James A. Robinson, 2012. "The Colonial Origins of Comparative Development: An Empirical Investigation: Reply," American Economic Review, American Economic Association, vol. 102(6), pages 3077-3110, October.
    28. Richard Jong-A-Pin & Jakob De Haan, 2008. "Growth Accelerations and Regime Changes: A Correction," Econ Journal Watch, Econ Journal Watch, vol. 5(1), pages 51-58, January.
    29. John Gibson & David L. Anderson & John Tressler, 2014. "Which Journal Rankings Best Explain Academic Salaries? Evidence From The University Of California," Economic Inquiry, Western Economic Association International, vol. 52(4), pages 1322-1340, October.
    30. Donald R. Winkler, 1975. "Educational Achievement and School Peer Group Composition," Journal of Human Resources, University of Wisconsin Press, vol. 10(2), pages 189-204.
    31. James Alm, 2010. "A Call for Replication Studies," Public Finance Review, , vol. 38(4), pages 501-507, July.
    32. McCullough, B. D. & McGeary, Kerry Anne & Harrison, Teresa D., 2006. "Lessons from the JMCB Archive," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 38(4), pages 1093-1107, June.
    33. Ashenfelter, Orley & Harmon, Colm & Oosterbeek, Hessel, 1999. "A review of estimates of the schooling/earnings relationship, with tests for publication bias," Labour Economics, Elsevier, vol. 6(4), pages 453-470, November.
    34. Caroline M. Hoxby, 2007. "Does Competition Among Public Schools Benefit Students and Taxpayers? Reply," American Economic Review, American Economic Association, vol. 97(5), pages 2038-2055, December.
    35. Andreas Ortmann & John Fitzgerald & Carl Boeing, 2000. "Trust, Reciprocity, and Social History: A Re-examination," Experimental Economics, Springer;Economic Science Association, vol. 3(1), pages 81-100, June.
    36. Basu, Kaushik & Narayan, Ambar & Ravallion, Martin, 2001. "Is literacy shared within households? Theory and evidence for Bangladesh," Labour Economics, Elsevier, vol. 8(6), pages 649-665, December.
    37. Lant Pritchett, 2002. "It pays to be ignorant: A simple political economy of rigorous program evaluation," Journal of Economic Policy Reform, Taylor & Francis Journals, vol. 5(4), pages 251-269.
    38. Jesse Rothstein, 2007. "Does Competition Among Public Schools Benefit Students and Taxpayers? Comment," American Economic Review, American Economic Association, vol. 97(5), pages 2026-2037, December.
    39. Mayer, Thomas, 1980. "Economics as a Hard Science: Realistic Goal or Wishful Thinking?," Economic Inquiry, Western Economic Association International, vol. 18(2), pages 165-178, April.
    40. John Ioannidis & Chris Doucouliagos, 2013. "What'S To Know About The Credibility Of Empirical Economics?," Journal of Economic Surveys, Wiley Blackwell, vol. 27(5), pages 997-1004, December.
    41. Zacharias Maniadis & Fabio Tufano & John A. List, 2014. "One Swallow Doesn't Make a Summer: New Evidence on Anchoring Effects," American Economic Review, American Economic Association, vol. 104(1), pages 277-290, January.
    42. James Alm, 2010. "A Call for Replication Studies," Public Finance Review, , vol. 38(5), pages 647-653, September.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Benedikt Fecher & Mathis Fräßdorf & Gert G. Wagner, 2016. "Perceptions and Practices of Replication by Social and Behavioral Scientists: Making Replications a Mandatory Element of Curricula Would Be Useful," Discussion Papers of DIW Berlin 1572, DIW Berlin, German Institute for Economic Research.
    2. Frank Mueller-Langer & Benedikt Fecher & Dietmar Harhoff & Gert G. Wagner, 2017. "The Economics of Replication," Discussion Papers of DIW Berlin 1640, DIW Berlin, German Institute for Economic Research.
    3. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," EconStor Open Access Articles, ZBW - Leibniz Information Centre for Economics, pages 62-83.
    4. Maren Duvendack & Richard Palmer-Jones & W. Robert Reed, 2017. "What Is Meant by "Replication" and Why Does It Encounter Resistance in Economics?," American Economic Review, American Economic Association, vol. 107(5), pages 46-51, May.
    5. Lehberger, Mira & Gruener, Sven, 2020. "Consumers’ willingness to pay for plants protected by beneficial insects—Evidence from two stated-choice experiments with different subject pools," SocArXiv 5zc62, Center for Open Science.
    6. W. Robert Reed, 2017. "Replication in labor economics," IZA World of Labor, Institute of Labor Economics (IZA), pages 413-413, December.
    7. Christian Zimmermann, 2015. "On the Need for a Replication Journal," Working Papers 2015-16, Federal Reserve Bank of St. Louis.
    8. W. Robert Reed, 2018. "A Primer on the ‘Reproducibility Crisis’ and Ways to Fix It," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 51(2), pages 286-300, June.
    9. Hernández Alemán, Anastasia & León, Carmelo J., 2018. "La Réplica en el Análisis Económico Aplicado/Replication in Applied Economic Analysis," Estudios de Economia Aplicada, Estudios de Economia Aplicada, vol. 36, pages 317-332, Enero.
    10. Vlaeminck, Sven & Podkrajac, Felix, 2017. "Journals in Economic Sciences: Paying Lip Service to Reproducible Research?," EconStor Open Access Articles, ZBW - Leibniz Information Centre for Economics, pages 1-16.
    11. Nicolas Vallois & Dorian Jullien, 2017. "Replication in experimental economics: A historical and quantitative approach focused on public good game experiments," Working Papers halshs-01651080, HAL.
    12. Hirschauer Norbert & Grüner Sven & Mußhoff Oliver & Frey Ulrich & Theesfeld Insa & Wagner Peter, 2016. "Die Interpretation des p-Wertes – Grundsätzliche Missverständnisse," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 236(5), pages 557-575, October.
    13. Nicolas Vallois & Dorian Jullien, 2017. "Replication in experimental economics: A historical and quantitative approach focused on public good game experiments," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) halshs-01651080, HAL.
    14. Zacharias Maniadis & Fabio Tufano & John A. List, 2017. "To Replicate or Not to Replicate? Exploring Reproducibility in Economics through the Lens of a Model and a Pilot Study," Economic Journal, Royal Economic Society, vol. 127(605), pages 209-235.
    15. Nicolas Vallois & Dorian Jullien, 2017. "Replication in Experimental Economics: A Historical and Quantitative Approach Focused on Public Good Game Experiments," GREDEG Working Papers 2017-21, Groupe de REcherche en Droit, Economie, Gestion (GREDEG CNRS), Université Côte d'Azur, France.
    16. Leiva, Benjamin & Liu, Zhongyuan, 2019. "Energy and economic growth in the USA two decades later: Replication and reanalysis," Energy Economics, Elsevier, vol. 82(C), pages 89-99.
    17. Bruns, Stephan B. & König, Johannes & Stern, David I., 2019. "Replication and robustness analysis of ‘energy and economic growth in the USA: A multivariate approach’," Energy Economics, Elsevier, vol. 82(C), pages 100-113.
    18. Paldam, Martin, 2018. "A model of the representative economist, as researcher and policy advisor," European Journal of Political Economy, Elsevier, vol. 54(C), pages 5-15.
    19. Winker Peter, 2016. "Editorial Announcement," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 236(1), pages 1-2, February.

    More about this item

    Keywords

    Replication; data sharing; publication bias;

    JEL classification:

    • A1 - General Economics and Teaching - - General Economics
    • B4 - Schools of Economic Thought and Methodology - - Economic Methodology

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ejw:journl:v:12:y:2015:i:2:p:164-191. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Jason Briggeman). General contact details of provider: http://edirc.repec.org/data/edgmuus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.