IDEAS home Printed from https://ideas.repec.org/p/cbt/econwp/14-26.html
   My bibliography  Save this paper

Replications in Economics: A Progress Report

Author

Listed:

Abstract

This study reports on various aspects of replication research in economics. It includes (i) a brief history of data sharing and replication; (ii) the results of the authors’ survey administered to the editors of all 333 “Economics” journals listed in Web of Science in December 2013; (iii) an analysis of 155 replication studies that have been published in peer-reviewed economics journals from 1977-2014; (iv) a discussion of the future of replication research in economics, and (v) observations on how replications can be better integrated into research efforts to address problems associated with publication bias and other Type I error phenomena.

Suggested Citation

  • Maren Duvendack & Richard W. Palmer-Jones & W. Robert Reed, 2014. "Replications in Economics: A Progress Report," Working Papers in Economics 14/26, University of Canterbury, Department of Economics and Finance.
  • Handle: RePEc:cbt:econwp:14/26
    as

    Download full text from publisher

    File URL: https://repec.canterbury.ac.nz/cbt/econwp/1426.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Siskind, Frederic B, 1977. "Minimum Wage Legislation in the United States: Comment," Economic Inquiry, Western Economic Association International, vol. 15(1), pages 135-138, January.
    2. Marshall S. Smith, 1968. "Equality of Educational Opportunity: Comments on Bowles and Levin," Journal of Human Resources, University of Wisconsin Press, vol. 3(3), pages 384-389.
    3. Richard Anderson & William Greene & B. D. McCullough & H. D. Vinod, 2008. "The role of data/code archives in the future of economic research," Journal of Economic Methodology, Taylor & Francis Journals, vol. 15(1), pages 99-119.
    4. Charles R. Link & Edward C. Ratledge, 1975. "Social Returns to Quantity and Quality of Education: A Further Statement," Journal of Human Resources, University of Wisconsin Press, vol. 10(1), pages 78-89.
    5. T. D. Stanley & Stephen B. Jarrell, 2005. "Meta‐Regression Analysis: A Quantitative Method of Literature Surveys," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 299-308, July.
    6. Jan H. Höffler, 2014. "Teaching Replication in Quantitative Empirical Economics," Replication Working Papers 2/2014, Institut für Statistik und Ökonometrie, Wirtschaftswissenschaftliche Fakultät, Georg-August-Universität Göttingen, Replication project.
    7. Stan J. Liebowitz, 2014. "Willful Blindness: The Inefficient Reward Structure In Academic Research," Economic Inquiry, Western Economic Association International, vol. 52(4), pages 1267-1283, October.
    8. Vegard Iversen & Richard Palmer-Jones, 2008. "Literacy Sharing, Assortative Mating, or What? Labour Market Advantages and Proximate Illiteracy Revisited," Journal of Development Studies, Taylor & Francis Journals, vol. 44(6), pages 797-838.
    9. B. D. McCullough & H. D. Vinod, 2003. "Verifying the Solution from a Nonlinear Solver: A Case Study," American Economic Review, American Economic Association, vol. 93(3), pages 873-892, June.
    10. Card, David & Krueger, Alan B, 1995. "Time-Series Minimum-Wage Studies: A Meta-analysis," American Economic Review, American Economic Association, vol. 85(2), pages 238-243, May.
    11. David Y. Albouy, 2012. "The Colonial Origins of Comparative Development: An Empirical Investigation: Comment," American Economic Review, American Economic Association, vol. 102(6), pages 3059-3076, October.
    12. Lowell E. Gallaway & Zachary Dyckman, 1970. "The Full Employment-Unemployment Rate: 1953-1980," Journal of Human Resources, University of Wisconsin Press, vol. 5(4), pages 487-510.
    13. Laura Camfield & Richard Palmer-Jones, 2013. "Three 'Rs' of Econometrics: Repetition, Reproduction and Replication," Journal of Development Studies, Taylor & Francis Journals, vol. 49(12), pages 1607-1614, December.
    14. Baltagi, Badi H & Khanti-Akom, Sophon, 1990. "On Efficient Estimation with Panel Data: An Empirical Comparison of Instrumental Variables Estimators," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 5(4), pages 401-406, Oct.-Dec..
    15. Robert Jensen & Emily Oster, 2009. "The Power of TV: Cable Television and Women's Status in India," The Quarterly Journal of Economics, Oxford University Press, vol. 124(3), pages 1057-1094.
    16. Leonard E. Burman & W. Robert Reed & James Alm, 2010. "A Call for Replication Studies," Public Finance Review, , vol. 38(6), pages 787-793, November.
    17. Daron Acemoglu & Simon Johnson & James A. Robinson, 2012. "The Colonial Origins of Comparative Development: An Empirical Investigation: Reply," American Economic Review, American Economic Association, vol. 102(6), pages 3077-3110, October.
    18. Richard Jong-A-Pin & Jakob De Haan, 2008. "Growth Accelerations and Regime Changes: A Correction," Econ Journal Watch, Econ Journal Watch, vol. 5(1), pages 51-58, January.
    19. John Gibson & David L. Anderson & John Tressler, 2014. "Which Journal Rankings Best Explain Academic Salaries? Evidence From The University Of California," Economic Inquiry, Western Economic Association International, vol. 52(4), pages 1322-1340, October.
    20. Donald R. Winkler, 1975. "Educational Achievement and School Peer Group Composition," Journal of Human Resources, University of Wisconsin Press, vol. 10(2), pages 189-204.
    21. James Alm, 2010. "A Call for Replication Studies," Public Finance Review, , vol. 38(4), pages 501-507, July.
    22. John S. Akin & Thomas J. Kniesner, 1976. "Proxies for Observations on Individuals Sampled from a Population," Journal of Human Resources, University of Wisconsin Press, vol. 11(3), pages 411-413.
    23. David W. Findlay & John M. Santos, 2012. "Race, Ethnicity, and Baseball Card Prices: A Replication, Correction, and Extension of Hewitt, Muñoz, Oliver, and Regoli," Econ Journal Watch, Econ Journal Watch, vol. 9(2), pages 122-140, May.
    24. Orley Ashenfelter & Colm Harmon & Hessel Oosterbeek, 1999. "A Review of Estimates of the Schooling/Earnings Relationship, with Tests for Publication Bias," Working Papers 804, Princeton University, Department of Economics, Industrial Relations Section..
    25. Peter Gordon & Lanlan Wang, 2004. "Does Economic Performance Correlate with Big Government?," Econ Journal Watch, Econ Journal Watch, vol. 1(2), pages 192-221, August.
    26. McCullough, B. D. & McGeary, Kerry Anne & Harrison, Teresa D., 2006. "Lessons from the JMCB Archive," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 38(4), pages 1093-1107, June.
    27. Ashenfelter, Orley & Harmon, Colm & Oosterbeek, Hessel, 1999. "A review of estimates of the schooling/earnings relationship, with tests for publication bias," Labour Economics, Elsevier, vol. 6(4), pages 453-470, November.
    28. Caroline M. Hoxby, 2007. "Does Competition Among Public Schools Benefit Students and Taxpayers? Reply," American Economic Review, American Economic Association, vol. 97(5), pages 2038-2055, December.
    29. Andreas Ortmann & John Fitzgerald & Carl Boeing, 2000. "Trust, Reciprocity, and Social History: A Re-examination," Experimental Economics, Springer;Economic Science Association, vol. 3(1), pages 81-100, June.
    30. Basu, Kaushik & Narayan, Ambar & Ravallion, Martin, 2001. "Is literacy shared within households? Theory and evidence for Bangladesh," Labour Economics, Elsevier, vol. 8(6), pages 649-665, December.
    31. Lant Pritchett, 2002. "It pays to be ignorant: A simple political economy of rigorous program evaluation," Journal of Economic Policy Reform, Taylor & Francis Journals, vol. 5(4), pages 251-269.
    32. Jesse Rothstein, 2007. "Does Competition Among Public Schools Benefit Students and Taxpayers? Comment," American Economic Review, American Economic Association, vol. 97(5), pages 2026-2037, December.
    33. Mayer, Thomas, 1980. "Economics as a Hard Science: Realistic Goal or Wishful Thinking?," Economic Inquiry, Western Economic Association International, vol. 18(2), pages 165-178, April.
    34. Charles R. Link & Edward C. Ratledge, 1976. "Proxies for Observations on Individuals Sampled from a Population: Reply," Journal of Human Resources, University of Wisconsin Press, vol. 11(3), pages 413-419.
    35. Justin McCrary, 2002. "Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime: Comment," American Economic Review, American Economic Association, vol. 92(4), pages 1236-1243, September.
    36. John Ioannidis & Chris Doucouliagos, 2013. "What'S To Know About The Credibility Of Empirical Economics?," Journal of Economic Surveys, Wiley Blackwell, vol. 27(5), pages 997-1004, December.
    37. Zacharias Maniadis & Fabio Tufano & John A. List, 2014. "One Swallow Doesn't Make a Summer: New Evidence on Anchoring Effects," American Economic Review, American Economic Association, vol. 104(1), pages 277-290, January.
    38. Feige, Edgar L, 1975. "The Consequences of Journal Editorial Policies and a Suggestion for Revision," Journal of Political Economy, University of Chicago Press, vol. 83(6), pages 1291-1295, December.
    39. Maren Duvendack & Richard Palmer-Jones, 2013. "Replication of quantitative work in development studies: Experiences and suggestions," Progress in Development Studies, , vol. 13(4), pages 307-322, October.
    40. James Alm, 2010. "A Call for Replication Studies," Public Finance Review, , vol. 38(5), pages 647-653, September.
    41. repec:fth:prinin:425 is not listed on IDEAS
    42. Chris Doucouliagos, 2005. "Publication Bias in the Economic Freedom and Economic Growth Literature," Journal of Economic Surveys, Wiley Blackwell, vol. 19(3), pages 367-387, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Benedikt Fecher & Mathis Fräßdorf & Gert G. Wagner, 2016. "Perceptions and Practices of Replication by Social and Behavioral Scientists: Making Replications a Mandatory Element of Curricula Would Be Useful," Discussion Papers of DIW Berlin 1572, DIW Berlin, German Institute for Economic Research.
    2. Frank Mueller-Langer & Benedikt Fecher & Dietmar Harhoff & Gert G. Wagner, 2017. "The Economics of Replication," Discussion Papers of DIW Berlin 1640, DIW Berlin, German Institute for Economic Research.
    3. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, pages 62-83.
    4. Nicolas Vallois & Dorian Jullien, 2017. "Replication in Experimental Economics: A Historical and Quantitative Approach Focused on Public Good Game Experiments," GREDEG Working Papers 2017-21, Groupe de REcherche en Droit, Economie, Gestion (GREDEG CNRS), Université Côte d'Azur, France.
    5. Maren Duvendack & Richard Palmer-Jones & W. Robert Reed, 2017. "What Is Meant by "Replication" and Why Does It Encounter Resistance in Economics?," American Economic Review, American Economic Association, vol. 107(5), pages 46-51, May.
    6. W. Robert Reed, 2017. "Replication in labor economics," IZA World of Labor, Institute of Labor Economics (IZA), pages 413-413, December.
    7. Emma McManus & David Turner & Tracey Sach, 2019. "Can You Repeat That? Exploring the Definition of a Successful Model Replication in Health Economics," PharmacoEconomics, Springer, vol. 37(11), pages 1371-1381, November.
    8. Christophe Hurlin & Christophe Pérignon, 2020. "Reproducibility Certification in Economics Research," Working Papers hal-02896404, HAL.
    9. Leiva, Benjamin & Liu, Zhongyuan, 2019. "Energy and economic growth in the USA two decades later: Replication and reanalysis," Energy Economics, Elsevier, vol. 82(C), pages 89-99.
    10. Zacharias Maniadis & Fabio Tufano & John A. List, 2017. "To Replicate or Not To Replicate? Exploring Reproducibility in Economics through the Lens of a Model and a Pilot Study," Economic Journal, Royal Economic Society, vol. 127(605), pages 209-235, October.
    11. Lehberger, Mira & Grüner, Sven, 2021. "Consumers’ willingness to pay for plants protected by beneficial insects – Evidence from two stated-choice experiments with different subject pools," Food Policy, Elsevier, vol. 102(C).
    12. Paldam, Martin, 2018. "A model of the representative economist, as researcher and policy advisor," European Journal of Political Economy, Elsevier, vol. 54(C), pages 5-15.
    13. Hirschauer Norbert & Grüner Sven & Mußhoff Oliver & Becker Claudia, 2019. "Twenty Steps Towards an Adequate Inferential Interpretation of p-Values in Econometrics," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 239(4), pages 703-721, August.
    14. Christian Zimmermann, 2015. "On the Need for a Replication Journal," Working Papers 2015-16, Federal Reserve Bank of St. Louis.
    15. Winker Peter, 2016. "Editorial Announcement," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 236(1), pages 1-2, February.
    16. W. Robert Reed, 2018. "A Primer on the ‘Reproducibility Crisis’ and Ways to Fix It," Australian Economic Review, The University of Melbourne, Melbourne Institute of Applied Economic and Social Research, vol. 51(2), pages 286-300, June.
    17. Hernández Alemán, Anastasia & León, Carmelo J., 2018. "La Réplica en el Análisis Económico Aplicado/Replication in Applied Economic Analysis," Estudios de Economia Aplicada, Estudios de Economia Aplicada, vol. 36, pages 317-332, Enero.
    18. Vlaeminck, Sven & Podkrajac, Felix, 2017. "Journals in Economic Sciences: Paying Lip Service to Reproducible Research?," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, pages 1-16.
    19. Bruns, Stephan B. & König, Johannes & Stern, David I., 2019. "Replication and robustness analysis of ‘energy and economic growth in the USA: A multivariate approach’," Energy Economics, Elsevier, vol. 82(C), pages 100-113.
    20. Nicolas Vallois & Dorian Jullien, 2017. "Replication in experimental economics: A historical and quantitative approach focused on public good game experiments," Working Papers halshs-01651080, HAL.
    21. Hirschauer Norbert & Grüner Sven & Mußhoff Oliver & Frey Ulrich & Theesfeld Insa & Wagner Peter, 2016. "Die Interpretation des p-Wertes – Grundsätzliche Missverständnisse," Journal of Economics and Statistics (Jahrbuecher fuer Nationaloekonomie und Statistik), De Gruyter, vol. 236(5), pages 557-575, October.
    22. Nicolas Vallois & Dorian Jullien, 2017. "Replication in experimental economics: A historical and quantitative approach focused on public good game experiments," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) halshs-01651080, HAL.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Christensen, Garret & Miguel, Edward & Sturdy, Jennifer, 2017. "Transparency, Reproducibility, and the Credibility of Economics Research," MetaArXiv 9a3rw, Center for Open Science.
    2. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    3. Christopher Snyder & Ran Zhuo, 2018. "Sniff Tests as a Screen in the Publication Process: Throwing out the Wheat with the Chaff," NBER Working Papers 25058, National Bureau of Economic Research, Inc.
    4. Hristos Doucouliagos & T. D. Stanley, 2009. "Publication Selection Bias in Minimum‐Wage Research? A Meta‐Regression Analysis," British Journal of Industrial Relations, London School of Economics, vol. 47(2), pages 406-428, June.
    5. Michael A. Clemens, 2017. "The Meaning Of Failed Replications: A Review And Proposal," Journal of Economic Surveys, Wiley Blackwell, vol. 31(1), pages 326-342, February.
    6. Maria Cipollina & Luca Salvatici, 2010. "Reciprocal Trade Agreements in Gravity Models: A Meta‐Analysis," Review of International Economics, Wiley Blackwell, vol. 18(1), pages 63-80, February.
    7. Fidrmuc, Jarko & Lind, Ronja, 2020. "Macroeconomic impact of Basel III: Evidence from a meta-analysis," Journal of Banking & Finance, Elsevier, vol. 112(C).
    8. Stanley, T. D. & Doucouliagos, Chris, 2019. "Practical Significance, Meta-Analysis and the Credibility of Economics," IZA Discussion Papers 12458, Institute of Labor Economics (IZA).
    9. Sebri, Maamar, 2015. "Use renewables to be cleaner: Meta-analysis of the renewable energy consumption–economic growth nexus," Renewable and Sustainable Energy Reviews, Elsevier, vol. 42(C), pages 657-665.
    10. Ian Babetskii & Nauro Campos, 2006. "Does Reform Work? An Econometric Examination of the Reform-Growth Puzzle," William Davidson Institute Working Papers Series wp870, William Davidson Institute at the University of Michigan.
    11. Neves, Pedro Cunha & Afonso, Óscar & Silva, Sandra Tavares, 2016. "A Meta-Analytic Reassessment of the Effects of Inequality on Growth," World Development, Elsevier, vol. 78(C), pages 386-400.
    12. Crespo Cuaresma, Jesús & Fidrmuc, Jarko & Hake, Mariya, 2014. "Demand and supply drivers of foreign currency loans in CEECs: A meta-analysis," Economic Systems, Elsevier, vol. 38(1), pages 26-42.
    13. Hamdaoui, Mekki, 2017. "Financial liberalization and systemic banking crises: A meta-analysis," International Economics, Elsevier, vol. 152(C), pages 26-54.
    14. Yong Yang & Nigel Driffield, 2012. "Multinationality-Performance Relationship," Management International Review, Springer, vol. 52(1), pages 23-47, February.
    15. Permani Risti, 2011. "The Impacts of Trade Liberalisation and Technological Change on GDP Growth in Indonesia: A Meta Regression Analysis," Global Economy Journal, De Gruyter, vol. 11(4), pages 1-30, December.
    16. Patricia C Melo & Yakubu Abdul-Salam & Deborah Roberts & Alana Gilbert & Robin Matthews & Liesbeth Colen & Sergio Gomez Y Paloma, 2015. "Income Elasticities of Food Demand in Africa: A Meta-Analysis," JRC Working Papers JRC98812, Joint Research Centre (Seville site).
    17. Jarko Fidrmuc & Katarína Danišková, 2020. "Meta-Analysis of the New Keynesian Phillips Curve in Developed and Emerging Economies," Emerging Markets Finance and Trade, Taylor & Francis Journals, vol. 56(1), pages 10-31, January.
    18. Liu, Boying & Richard Shumway, C., 2016. "Substitution elasticities between GHG-polluting and nonpolluting inputs in agricultural production: A meta-regression," Energy Economics, Elsevier, vol. 54(C), pages 123-132.
    19. Vlaeminck, Sven, 2013. "Data Management in Scholarly Journals and Possible Roles for Libraries - Some Insights from EDaWaX," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, pages 49-79.
    20. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.

    More about this item

    Keywords

    Replication; data sharing; publication bias;
    All these keywords.

    JEL classification:

    • A1 - General Economics and Teaching - - General Economics
    • B4 - Schools of Economic Thought and Methodology - - Economic Methodology

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cbt:econwp:14/26. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/decannz.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Albert Yee (email available below). General contact details of provider: https://edirc.repec.org/data/decannz.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.