IDEAS home Printed from https://ideas.repec.org/a/sae/prodev/v13y2013i4p307-322.html
   My bibliography  Save this article

Replication of quantitative work in development studies: Experiences and suggestions

Author

Listed:
  • Maren Duvendack

    (Lecturer, University of East Anglia, UK)

  • Richard Palmer-Jones

    (Research Fellow, University of East Anglia, UK)

Abstract

There is a growing demand for replications of authoritative works in development studies, which reflects recent trends in other social sciences as well as challenges to important quantitative works in development studies where replications have made contested contributions to understanding. At the same time, there is a strong trend within development towards adoption of medical models of evidence-based policy to find out what policies and interventions work. Replication is a key practice of medical (and natural science) research and was advocated frequently over several decades without success. This article addresses the incentives for replication going beyond a narrow focus on extrinsic rewards, reviews some significant examples, discusses behaviour during replication and draws lessons for replicators and replicatees.

Suggested Citation

  • Maren Duvendack & Richard Palmer-Jones, 2013. "Replication of quantitative work in development studies: Experiences and suggestions," Progress in Development Studies, , vol. 13(4), pages 307-322, October.
    • Handle: RePEc:sae:prodev:v:13:y:2013:i:4:p:307-322
    DOI: 10.1177/1464993413490480
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/1464993413490480
    Download Restriction: no
    File URL: https://libkey.io/10.1177/1464993413490480?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Jere R. Behrman & Mark R. Rosenzweig, 2005. "Does Increasing Women's Schooling Raise the Schooling of the Next Generation? Reply," American Economic Review, American Economic Association, vol. 95(5), pages 1745-1751, December.
    2. Stefano DellaVigna & Matthew Gentzkow, 2010. "Persuasion: Empirical Evidence," Annual Review of Economics, Annual Reviews, vol. 2(1), pages 643-669, September.
    3. Richard Anderson & William Greene & B. D. McCullough & H. D. Vinod, 2008. "The role of data/code archives in the future of economic research," Journal of Economic Methodology, Taylor & Francis Journals, vol. 15(1), pages 99-119.
    4. Steven D. Levitt, 2002. "Using Electoral Cycles in Police Hiring to Estimate the Effects of Police on Crime: Reply," American Economic Review, American Economic Association, vol. 92(4), pages 1244-1250, September.
    5. Matthieu Chemin, 2012. "Response to ‘High Noon for Microfinance Impact Evaluations’," Journal of Development Studies, Taylor & Francis Journals, vol. 48(12), pages 1881-1885, December.
    6. John P A Ioannidis, 2005. "Why Most Published Research Findings Are False," PLOS Medicine, Public Library of Science, vol. 2(8), pages 1-1, August.
    7. Daron Acemoglu & Simon Johnson & James A. Robinson, 2001. "The Colonial Origins of Comparative Development: An Empirical Investigation," American Economic Review, American Economic Association, vol. 91(5), pages 1369-1401, December.
    8. Dewald, William G & Thursby, Jerry G & Anderson, Richard G, 1986. "Replication in Empirical Economics: The Journal of Money, Credit and Banking Project," American Economic Review, American Economic Association, vol. 76(4), pages 587-603, September.
    9. Henrik Hansen & Finn Tarp, 2000. "Aid effectiveness disputed," Journal of International Development, John Wiley & Sons, Ltd., vol. 12(3), pages 375-398, April.
    10. Leimer, Dean R & Lesnoy, Selig D, 1982. "Social Security and Private Saving: New Time-Series Evidence," Journal of Political Economy, University of Chicago Press, vol. 90(3), pages 606-629, June.
    11. B. D. McCullough & H. D. Vinod, 2003. "Verifying the Solution from a Nonlinear Solver: A Case Study," American Economic Review, American Economic Association, vol. 93(3), pages 873-892, June.
    12. Kate L. Antonovics & Arthur S. Goldberger, 2005. "Does Increasing Women's Schooling Raise the Schooling of the Next Generation? Comment," American Economic Review, American Economic Association, vol. 95(5), pages 1738-1744, December.
    13. Charles F. Manski, 2011. "Policy Analysis with Incredible Certitude," Economic Journal, Royal Economic Society, vol. 121(554), pages 261-289, August.
    14. Albouy, David, 2006. "The Colonial Origins of Comparative Development: An Investigation of the Settler Mortality Data," Center for International and Development Economics Research, Working Paper Series qt8kt576x8, Center for International and Development Economics Research, Institute for Business and Economic Research, UC Berkeley.
    15. David Y. Albouy, 2012. "The Colonial Origins of Comparative Development: An Empirical Investigation: Comment," American Economic Review, American Economic Association, vol. 102(6), pages 3059-3076, October.
    16. Matthieu Chemin, 2008. "The Benefits and Costs of Microfinance: Evidence from Bangladesh," Journal of Development Studies, Taylor & Francis Journals, vol. 44(4), pages 463-484, April.
    17. Ted Joyce, 2004. "Did Legalized Abortion Lower Crime?," Journal of Human Resources, University of Wisconsin Press, vol. 39(1).
    18. Daniel S. Hamermesh, 2007. "Viewpoint: Replication in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 40(3), pages 715-733, August.
    19. Jere R. Behrman & Mark R. Rosenzweig, 2002. "Does Increasing Women's Schooling Raise the Schooling of the Next Generation?," American Economic Review, American Economic Association, vol. 92(1), pages 323-334, March.
    20. Daron Acemoglu & Simon Johnson & James A. Robinson, 2012. "The Colonial Origins of Comparative Development: An Empirical Investigation: Reply," American Economic Review, American Economic Association, vol. 102(6), pages 3077-3110, October.
    21. Caroline M. Hoxby, 2000. "The Effects of Class Size on Student Achievement: New Evidence from Population Variation," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 115(4), pages 1239-1285.
    22. Jonathan Schooler, 2011. "Unpublished results hide the decline effect," Nature, Nature, vol. 470(7335), pages 437-437, February.
    23. Levitt, Steven D, 1997. "Using Electoral Cycles in Police Hiring to Estimate the Effect of Police on Crime," American Economic Review, American Economic Association, vol. 87(3), pages 270-290, June.
    24. Martin Feldstein, 1982. "The Welfare Cost of Social Security's Impact on Private Saving," NBER Working Papers 0969, National Bureau of Economic Research, Inc.
    25. Leamer, Edward E, 1983. "Let's Take the Con Out of Econometrics," American Economic Review, American Economic Association, vol. 73(1), pages 31-43, March.
    26. Feldstein, Martin S, 1982. "Social Security and Private Saving: Reply," Journal of Political Economy, University of Chicago Press, vol. 90(3), pages 630-642, June.
    27. H. D. Vinod & B. D. McCullough, 1999. "The Numerical Reliability of Econometric Software," Journal of Economic Literature, American Economic Association, vol. 37(2), pages 633-665, June.
    28. Zeeya Merali, 2010. "Computational science: ...Error," Nature, Nature, vol. 467(7317), pages 775-777, October.
    29. B.D. McCullough & Kerry Anne McGeary & Teresa D. Harrison, 2008. "Do economics journal archives promote replicable research?," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 41(4), pages 1406-1420, November.
    30. Alan B. Krueger & David Card, 2000. "Minimum Wages and Employment: A Case Study of the Fast-Food Industry in New Jersey and Pennsylvania: Reply," American Economic Review, American Economic Association, vol. 90(5), pages 1397-1420, December.
    31. B.D. McCullough, 2009. "Open Access Economics Journals and the Market for Reproducible Economic Research," Economic Analysis and Policy, Elsevier, vol. 39(1), pages 117-126, March.
    32. David F. Hendry & Hans‐Martin Krolzig, 2004. "We Ran One Regression," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 66(5), pages 799-810, December.
    33. David F. Hendry & Hans-Martin Krolzig, 2004. "We Ran One Regression," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 66(5), pages 799-810, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    2. Wood, Benjamin Douglas Kuflick & Vasquez, Maria, 2018. "Microplots and food security: Encouraging replication studies of policy relevant research," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 12, pages 1-12.
    3. Maren Duvendack & Richard Palmer-Jones & W. Robert Reed, 2017. "What Is Meant by "Replication" and Why Does It Encounter Resistance in Economics?," American Economic Review, American Economic Association, vol. 107(5), pages 46-51, May.
    4. Maren Duvendack & Richard W. Palmer-Jones & W. Robert Reed, 2015. "Replications in Economics: A Progress Report," Econ Journal Watch, Econ Journal Watch, vol. 12(2), pages 164–191-1, May.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Michael A. Clemens, 2017. "The Meaning Of Failed Replications: A Review And Proposal," Journal of Economic Surveys, Wiley Blackwell, vol. 31(1), pages 326-342, February.
    2. Daniel S. Hamermesh, 2007. "Viewpoint: Replication in economics," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 40(3), pages 715-733, August.
    3. Duvendack, Maren & Palmer-Jones, Richard, 2011. "The microfinance of reproduction and the reproduction of microfinance: understanding the connections between microfinance, empowerment, contraception and fertility in Bangladesh in the 1990s," MPRA Paper 32384, University Library of Munich, Germany.
    4. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    5. Valérie Orozco & Christophe Bontemps & Elise Maigné & Virginie Piguet & Annie Hofstetter & Anne Lacroix & Fabrice Levert & Jean‐Marc Rousselle, 2020. "How To Make A Pie: Reproducible Research For Empirical Economics And Econometrics," Journal of Economic Surveys, Wiley Blackwell, vol. 34(5), pages 1134-1169, December.
    6. Mark J. McCabe & Frank Mueller-Langer, 2019. "Does Data Disclosure Increase Citations? Empirical Evidence from a Natural Experiment in Leading Economics Journals," JRC Working Papers on Digital Economy 2019-02, Joint Research Centre.
    7. Maren Duvendack & Richard W. Palmer-Jones & W. Robert Reed, 2015. "Replications in Economics: A Progress Report," Econ Journal Watch, Econ Journal Watch, vol. 12(2), pages 164–191-1, May.
    8. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," Research Policy, Elsevier, vol. 48(1), pages 62-83.
    9. Andrew C. Chang & Phillip Li, 2015. "Is Economics Research Replicable? Sixty Published Papers from Thirteen Journals Say \"Usually Not\"," Finance and Economics Discussion Series 2015-83, Board of Governors of the Federal Reserve System (U.S.).
    10. Hamermesh, Daniel S., 2007. "Replication in Economics," IZA Discussion Papers 2760, Institute of Labor Economics (IZA).
    11. Vlaeminck, Sven, 2013. "Data Management in Scholarly Journals and Possible Roles for Libraries - Some Insights from EDaWaX," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 23(1), pages 49-79.
    12. Charles G. Renfro, 2009. "The Practice of Econometric Theory," Advanced Studies in Theoretical and Applied Econometrics, Springer, number 978-3-540-75571-5, July-Dece.
    13. Horton, Joanne & Krishna Kumar, Dhanya & Wood, Anthony, 2020. "Detecting academic fraud using Benford law: The case of Professor James Hunton," Research Policy, Elsevier, vol. 49(8).
    14. Morduch, Jonathan & Ravi, Shamika & Bauchet, Jonathan, 2013. "Substitution Bias and External Validity: Why an Innovative Anti-poverty Program Showed no Net Impact," CEI Working Paper Series 2013-02, Center for Economic Institutions, Institute of Economic Research, Hitotsubashi University.
    15. Maren Duvendack & Richard Palmer-Jones, 2012. "High Noon for Microfinance Impact Evaluations: Re-investigating the Evidence from Bangladesh," Journal of Development Studies, Taylor & Francis Journals, vol. 48(12), pages 1864-1880, December.
    16. Mueller-Langer, Frank & Andreoli-Versbach, Patrick, 2018. "Open access to research data: Strategic delay and the ambiguous welfare effects of mandatory data disclosure," Information Economics and Policy, Elsevier, vol. 42(C), pages 20-34.
    17. Andreoli-Versbach, Patrick & Mueller-Langer, Frank, 2014. "Open access to data: An ideal professed but not practised," Research Policy, Elsevier, vol. 43(9), pages 1621-1633.
    18. Carlisle E. Moody & Thomas B. Marvell, 2010. "On the Choice of Control Variables in the Crime Equation," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 72(5), pages 696-715, October.
    19. Abel Brodeur & Mathias Lé & Marc Sangnier & Yanos Zylberberg, 2016. "Star Wars: The Empirics Strike Back," American Economic Journal: Applied Economics, American Economic Association, vol. 8(1), pages 1-32, January.
    20. Valérie Orozco & Christophe Bontemps & Élise Maigné & Virginie Piguet & Annie Hofstetter & Anne Marie Lacroix & Fabrice Levert & Jean-Marc Rousselle, 2017. "How to make a pie? Reproducible Research for Empirical Economics & Econometrics," Post-Print hal-01939942, HAL.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:prodev:v:13:y:2013:i:4:p:307-322. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.