IDEAS home Printed from https://ideas.repec.org/a/taf/jdevef/v6y2014i3p215-235.html
   My bibliography  Save this article

Quality evidence for policymaking: I'll believe it when I see the replication

Author

Listed:
  • Annette N. Brown
  • Drew B. Cameron
  • Benjamin D. K. Wood

Abstract

In this paper, we make the case for replication as a crucial methodology for validating research used for evidence-based policymaking, especially in low- and middle-income countries. We focus on internal replication or the reanalysis of original data to address an original evaluation or research question. We review the current state of replication in the social sciences and present data on the trends among academic publications. We then discuss four challenges facing empirical research that internal replication can help to address. We offer a new typology of replication approaches for addressing these challenges. The types - pure replication, measurement and estimation analysis, and theory of change analysis - highlight that internal replication can test for consistency and statistical robustness but can and should also be used to ensure that a study fully explores possible theories of change in order to draw appropriate conclusions and recommendations for policymaking and programme design.

Suggested Citation

  • Annette N. Brown & Drew B. Cameron & Benjamin D. K. Wood, 2014. "Quality evidence for policymaking: I'll believe it when I see the replication," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(3), pages 215-235, September.
    • Handle: RePEc:taf:jdevef:v:6:y:2014:i:3:p:215-235
    DOI: 10.1080/19439342.2014.944555
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1080/19439342.2014.944555
    Download Restriction: Access to full text is restricted to subscribers.
    File URL: https://libkey.io/10.1080/19439342.2014.944555?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Angus Deaton, 2010. "Instruments, Randomization, and Learning about Development," Journal of Economic Literature, American Economic Association, vol. 48(2), pages 424-455, June.
    2. John J. Donohue III & Steven D. Levitt, 2008. "Measurement Error, Legalized Abortion, and the Decline in Crime: A Response to Foote and Goetz," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 123(1), pages 425-440.
    3. Abhijit V. Banerjee & Shawn Cole & Esther Duflo & Leigh Linden, 2007. "Remedying Education: Evidence from Two Randomized Experiments in India," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(3), pages 1235-1264.
    4. Andrew Abbott, 2007. "Notes on Replication," Sociological Methods & Research, , vol. 36(2), pages 210-219, November.
    5. Humphreys, Macartan & Sanchez de la Sierra, Raul & van der Windt, Peter, 2013. "Fishing, Commitment, and Communication: A Proposal for Comprehensive Nonbinding Research Registration," Political Analysis, Cambridge University Press, vol. 21(1), pages 1-20, January.
    6. Leonard E. Burman & W. Robert Reed & James Alm, 2011. "A Call for Replication Studies," Public Finance Review, , vol. 39(1), pages 190-190, January.
    7. Carmen M. Reinhart & Kenneth S. Rogoff, 2010. "Growth in a Time of Debt," American Economic Review, American Economic Association, vol. 100(2), pages 573-578, May.
    8. Joshua D. Angrist & Jörn-Steffen Pischke, 2010. "The Credibility Revolution in Empirical Economics: How Better Research Design Is Taking the Con out of Econometrics," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 3-30, Spring.
    9. Jeremy Freese, 2007. "Overcoming Objections to Open-Source Social Science," Sociological Methods & Research, , vol. 36(2), pages 220-226, November.
    10. John J. Donohue III & Steven D. Levitt, 2001. "The Impact of Legalized Abortion on Crime," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 116(2), pages 379-420.
    11. Evanschitzky, Heiner & Baumgarth, Carsten & Hubbard, Raymond & Armstrong, J. Scott, 2007. "Replication research's disturbing trend," Journal of Business Research, Elsevier, vol. 60(4), pages 411-415, April.
    12. Daniel S. Hamermesh, 2007. "Replication in Economics," NBER Working Papers 13026, National Bureau of Economic Research, Inc.
    13. Ed Yong, 2012. "Replication studies: Bad copy," Nature, Nature, vol. 485(7398), pages 298-300, May.
    14. Michael Carter & Christopher Barrett, 2006. "The economics of poverty traps and persistent poverty: An asset-based approach," Journal of Development Studies, Taylor & Francis Journals, vol. 42(2), pages 178-199.
    15. Edward Miguel & Michael Kremer, 2004. "Worms: Identifying Impacts on Education and Health in the Presence of Treatment Externalities," Econometrica, Econometric Society, vol. 72(1), pages 159-217, January.
    16. Binmore, Ken & Shaked, Avner, 2010. "Experimental economics: Where next?," Journal of Economic Behavior & Organization, Elsevier, vol. 73(1), pages 87-100, January.
    17. Pierre Berthon & Leyland Pitt & Michael Ewing & Christopher L. Carr, 2002. "Potential Research Space in MIS: A Framework for Envisioning and Evaluating Research Replication, Extension, and Generation," Information Systems Research, INFORMS, vol. 13(4), pages 416-427, December.
    18. Sachs, Jeffrey D & Warner, Andrew M, 1997. "Sources of Slow Growth in African Economies," Journal of African Economies, Centre for the Study of African Economies, vol. 6(3), pages 335-376, October.
    19. Daniel S. Hamermesh, 2007. "Viewpoint: Replication in economics," Canadian Journal of Economics, Canadian Economics Association, vol. 40(3), pages 715-733, August.
    20. Nancy Cartwright, 1991. "Replicability, Reproducibility, and Robustness: Comments on Harry Collins," History of Political Economy, Duke University Press, vol. 23(1), pages 143-155, Spring.
    21. Basu, Kaushik & Foster, James E, 1998. "On Measuring Literacy," Economic Journal, Royal Economic Society, vol. 108(451), pages 1733-1749, November.
    22. Feldstein, Martin S, 1982. "Social Security and Private Saving: Reply," Journal of Political Economy, University of Chicago Press, vol. 90(3), pages 630-642, June.
    23. Feldstein, Martin S, 1974. "Social Security, Induced Retirement, and Aggregate Capital Accumulation," Journal of Political Economy, University of Chicago Press, vol. 82(5), pages 905-926, Sept./Oct.
    24. Leimer, Dean R & Lesnoy, Selig D, 1982. "Social Security and Private Saving: New Time-Series Evidence," Journal of Political Economy, University of Chicago Press, vol. 90(3), pages 606-629, June.
    25. Lipset, Seymour Martin, 1959. "Some Social Requisites of Democracy: Economic Development and Political Legitimacy1," American Political Science Review, Cambridge University Press, vol. 53(1), pages 69-105, March.
    26. Binmore, Ken & Shaked, Avner, 2010. "Experimental Economics: Where Next? Rejoinder," Journal of Economic Behavior & Organization, Elsevier, vol. 73(1), pages 120-121, January.
    27. Martin Feldstein, 1982. "The Welfare Cost of Social Security's Impact on Private Saving," NBER Working Papers 0969, National Bureau of Economic Research, Inc.
    28. Karthik Muralidharan & Venkatesh Sundararaman, 2011. "Teacher Performance Pay: Experimental Evidence from India," Journal of Political Economy, University of Chicago Press, vol. 119(1), pages 39-77.
    29. Liverpool, Lenis Saweda O. & Winter-Nelson, Alex, 2010. "Asset versus consumption poverty and poverty dynamics in the presence of multiple equilibria in rural Ethiopia," IFPRI discussion papers 971, International Food Policy Research Institute (IFPRI).
    30. Edward E. Leamer, 2010. "Tantalus on the Road to Asymptopia," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 31-46, Spring.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Michael A. Clemens, 2017. "The Meaning Of Failed Replications: A Review And Proposal," Journal of Economic Surveys, Wiley Blackwell, vol. 31(1), pages 326-342, February.
    2. Benjamin D K Wood & Rui Müller & Annette N Brown, 2018. "Push button replication: Is impact evaluation evidence for international development verifiable?," PLOS ONE, Public Library of Science, vol. 13(12), pages 1-15, December.
    3. Wood, Benjamin Douglas Kuflick & Vasquez, Maria, 2018. "Microplots and food security: Encouraging replication studies of policy relevant research," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 12, pages 1-12.
    4. Brown, Annette N. & Wood, Benjamin Douglas Kuflick, 2018. "Which tests not witch hunts: A diagnostic approach for conducting replication research," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy (IfW Kiel), vol. 12, pages 1-26.
    5. Cameron, Lisa & Olivia, Susan & Shah, Manisha, 2019. "Scaling up sanitation: Evidence from an RCT in Indonesia," Journal of Development Economics, Elsevier, vol. 138(C), pages 1-16.
    6. Freese, Jeremy & Peterson, David, 2017. "Replication in Social Science," SocArXiv 5bck9, Center for Open Science.
    7. Donovan, Kevin P., 2018. "The rise of the randomistas: on the experimental turn in international aid," SocArXiv xygzb, Center for Open Science.
    8. Eric W Djimeu & Anna C Heard, 2019. "Treatment of HIV among tuberculosis patients: A replication study of timing of antiretroviral therapy for HIV-1-associated tuberculosis," PLOS ONE, Public Library of Science, vol. 14(2), pages 1-20, February.
    9. Emma McManus & David Turner & Tracey Sach, 2019. "Can You Repeat That? Exploring the Definition of a Successful Model Replication in Health Economics," PharmacoEconomics, Springer, vol. 37(11), pages 1371-1381, November.
    10. Cuong Viet Nguyen, 2016. "Impacts of rural road on household welfare in Vietnam: Evidence from a replication study," Economics Discussion Papers 2016-40, Kiel Institute for the World Economy (IfW Kiel).
    11. Wood, Benjamin D.K. & Dong, Michell, 2015. "Recalling Extra Data: A Replication Study of Finding Missing Markets," 2014: Food, Resources and Conflict, December 7-9, 2014. San Diego, California 206225, International Agricultural Trade Research Consortium.
    12. Eric W Djimeu & Anna Heard, 2020. "Replication of influential studies on biomedical, social, behavioural and structural interventions for HIV prevention and treatment," PLOS ONE, Public Library of Science, vol. 15(10), pages 1-10, October.
    13. Laura Outhwaite & Jake Anders & Jo Van Herwegen, 2022. "Mathematics Attainment Falls Behind Reading in the Early Primary School Years," CEPEO Working Paper Series 22-06, UCL Centre for Education Policy and Equalising Opportunities, revised May 2022.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    2. Michael A. Clemens, 2017. "The Meaning Of Failed Replications: A Review And Proposal," Journal of Economic Surveys, Wiley Blackwell, vol. 31(1), pages 326-342, February.
    3. Valérie Orozco & Christophe Bontemps & Elise Maigné & Virginie Piguet & Annie Hofstetter & Anne Lacroix & Fabrice Levert & Jean‐Marc Rousselle, 2020. "How To Make A Pie: Reproducible Research For Empirical Economics And Econometrics," Journal of Economic Surveys, Wiley Blackwell, vol. 34(5), pages 1134-1169, December.
    4. Maren Duvendack & Richard Palmer-Jones, 2013. "Replication of quantitative work in development studies: Experiences and suggestions," Progress in Development Studies, , vol. 13(4), pages 307-322, October.
    5. Baylis, Kathy & Ham, Andres, 2015. "How important is spatial correlation in randomized controlled trials?," 2015 AAEA & WAEA Joint Annual Meeting, July 26-28, San Francisco, California 205586, Agricultural and Applied Economics Association.
    6. Franklin Mixon, 2002. "Social security trust fund flows and the welfare costs of rent seeking," Applied Economics, Taylor & Francis Journals, vol. 34(8), pages 975-979.
    7. Karthik Muralidharan & Paul Niehaus, 2017. "Experimentation at Scale," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 103-124, Fall.
    8. Charles Yuji Horioka, 2014. "The Life and Work Of Martin Stuart (“Marty”) Feldstein," UP School of Economics Discussion Papers 201410, University of the Philippines School of Economics.
    9. Miguel Urquiola, 2015. "Progress and challenges in achieving an evidence-based education policy in Latin America and the Caribbean," Latin American Economic Review, Springer;Centro de Investigaciòn y Docencia Económica (CIDE), vol. 24(1), pages 1-30, December.
    10. Drazen, Allan & Dreber, Anna & Ozbay, Erkut Y. & Snowberg, Erik, 2021. "Journal-based replication of experiments: An application to “Being Chosen to Lead”," Journal of Public Economics, Elsevier, vol. 202(C).
    11. Bissonnette, L. & van Soest, A.H.O., 2010. "Retirement Expectations, Preferences, and Decisions," Other publications TiSEM 45e93b08-cc1d-47c6-ba06-d, Tilburg University, School of Economics and Management.
    12. Abhijit Banerjee & Rukmini Banerji & James Berry & Esther Duflo & Harini Kannan & Shobhini Mukerji & Marc Shotland & Michael Walton, 2017. "From Proof of Concept to Scalable Policies: Challenges and Solutions, with an Application," Journal of Economic Perspectives, American Economic Association, vol. 31(4), pages 73-102, Fall.
    13. Abhijit Banerjee & Sylvain Chassang & Erik Snowberg, 2016. "Decision Theoretic Approaches to Experiment Design and External Validity," NBER Working Papers 22167, National Bureau of Economic Research, Inc.
    14. Mueller-Langer, Frank & Fecher, Benedikt & Harhoff, Dietmar & Wagner, Gert G., 2019. "Replication studies in economics—How many and which papers are chosen for replication, and why?," Research Policy, Elsevier, vol. 48(1), pages 62-83.
    15. Martin Feldstein, 1985. "Should Social Security Be Means Tested?," NBER Working Papers 1775, National Bureau of Economic Research, Inc.
    16. Mark J. McCabe & Frank Mueller-Langer, 2019. "Does Data Disclosure Increase Citations? Empirical Evidence from a Natural Experiment in Leading Economics Journals," JRC Working Papers on Digital Economy 2019-02, Joint Research Centre.
    17. repec:dpr:wpaper:0905 is not listed on IDEAS
    18. Daniel S. Hamermesh, 2007. "Viewpoint: Replication in economics," Canadian Journal of Economics/Revue canadienne d'économique, John Wiley & Sons, vol. 40(3), pages 715-733, August.
    19. Charles S. Wassell, Jr., 2018. "Social Security and saving: A time-series econometrics pedagogical example (with code)," The Journal of Economic Education, Taylor & Francis Journals, vol. 49(1), pages 103-114, January.
    20. Donovan, Kevin P., 2018. "The rise of the randomistas: on the experimental turn in international aid," SocArXiv xygzb, Center for Open Science.
    21. Jonathan Gruber & Aaron Yelowitz, 1999. "Public Health Insurance and Private Savings," Journal of Political Economy, University of Chicago Press, vol. 107(6), pages 1249-1274, December.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:taf:jdevef:v:6:y:2014:i:3:p:215-235. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Longhurst (email available below). General contact details of provider: http://www.tandfonline.com/RJDE20 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.