IDEAS home Printed from https://ideas.repec.org/p/zbw/ifwedp/201777.html

Which tests not witch hunts: a diagnostic approach for conducting replication research

Author

Listed:
  • Brown, Annette N.
  • Wood, Benjamin Douglas Kuflick

Abstract

This paper provides researchers with an objective list of checks to consider when planning a replication study with the objective of validating findings for informing policy. These replication studies should begin with a pure replication of the published results and then reanalyse the original data to address the original research question. The author presents tips for replication exercises in four categories: validity of assumptions, data transformations, estimation methods, and heterogeneous impacts. For each category he offers an introduction, a tips checklist, some examples of how these checks have been employed, and a set of resources that provide statistical and econometric details.

Suggested Citation

  • Brown, Annette N. & Wood, Benjamin Douglas Kuflick, 2017. "Which tests not witch hunts: a diagnostic approach for conducting replication research," Economics Discussion Papers 2017-77, Kiel Institute for the World Economy.
    • Handle: RePEc:zbw:ifwedp:201777
    as

    Download full text from publisher

    File URL: http://www.economics-ejournal.org/economics/discussionpapers/2017-77
    Download Restriction: no
    File URL: https://www.econstor.eu/bitstream/10419/169136/1/898640105.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. W. Robert Reed, 2017. "Replication in labor economics," World of Labour, LISER, pages 413-413, December.
    2. Anderson, Michael L., 2008. "Multiple Inference and Gender Differences in the Effects of Early Intervention: A Reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects," Journal of the American Statistical Association, American Statistical Association, vol. 103(484), pages 1481-1495.
    3. Pradyot Ranjan Jena & Bezawit Beyene Chichaibelu & Till Stellmacher & Ulrike Grote, 2012. "The impact of coffee certification on small-scale producers’ livelihoods: a case study from the Jimma Zone, Ethiopia," Agricultural Economics, International Association of Agricultural Economists, vol. 43(4), pages 429-440, July.
    4. David McKenzie, 2017. "How Effective Are Active Labor Market Policies in Developing Countries? A Critical Review of Recent Evidence," The World Bank Research Observer, World Bank, vol. 32(2), pages 127-154.
    5. Cummins, Joseph R., 2017. "Heterogeneous treatment effects in the low track: Revisiting the Kenyan primary school experiment," Economics of Education Review, Elsevier, vol. 56(C), pages 40-51.
    6. Imai, Kosuke & Strauss, Aaron, 2011. "Estimation of Heterogeneous Treatment Effects from Randomized Experiments, with Application to the Optimal Planning of the Get-Out-the-Vote Campaign," Political Analysis, Cambridge University Press, vol. 19(1), pages 1-19, January.
    7. Stefan Dercon & Daniel O. Gilligan & John Hoddinott & Tassew Woldehanna, 2009. "The Impact of Agricultural Extension and Roads on Poverty and Consumption Growth in Fifteen Ethiopian Villages," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 91(4), pages 1007-1021.
    8. Kenneth F Schulz & Douglas G Altman & David Moher & for the CONSORT Group, 2010. "CONSORT 2010 Statement: Updated Guidelines for Reporting Parallel Group Randomised Trials," PLOS Medicine, Public Library of Science, vol. 7(3), pages 1-7, March.
    9. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    10. Raj Chetty & John N. Friedman & Jonah E. Rockoff, 2017. "Measuring the Impacts of Teachers: Reply," American Economic Review, American Economic Association, vol. 107(6), pages 1685-1717, June.
    11. David S. Lee & Enrico Moretti & Matthew J. Butler, 2004. "Do Voters Affect or Elect Policies? Evidence from the U. S. House," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 119(3), pages 807-859.
    12. Sebastian Galiani & Paul Gertler & Mauricio Romero, 2017. "Incentives for Replication in Economics," NBER Working Papers 23576, National Bureau of Economic Research, Inc.
    13. Benjamin D K Wood & Rui Müller & Annette N Brown, 2018. "Push button replication: Is impact evaluation evidence for international development verifiable?," PLOS ONE, Public Library of Science, vol. 13(12), pages 1-15, December.
    14. Jesse Rothstein, 2010. "Teacher Quality in Educational Production: Tracking, Decay, and Student Achievement," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 125(1), pages 175-214.
    15. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    16. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    17. Annette N. Brown & Drew B. Cameron & Benjamin D. K. Wood, 2014. "Quality evidence for policymaking: I'll believe it when I see the replication," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(3), pages 215-235, September.
    18. Christian Zimmermann, 2015. "On the Need for a Replication Journal," Working Papers 2015-16, Federal Reserve Bank of St. Louis.
    19. Galiani, Sebastian & Schargrodsky, Ernesto, 2010. "Property rights for the poor: Effects of land titling," Journal of Public Economics, Elsevier, vol. 94(9-10), pages 700-729, October.
    20. Matias D. Cattaneo & Sebastian Galiani & Paul J. Gertler & Sebastian Martinez & Rocio Titiunik, 2009. "Housing, Health, and Happiness," American Economic Journal: Economic Policy, American Economic Association, vol. 1(1), pages 75-105, February.
    21. Wood, Benjamin D.K. & Dong, Michell, 2015. "Recalling Extra Data: A Replication Study of Finding Missing Markets," 2014: Food, Resources and Conflict, December 7-9, 2014. San Diego, California 206225, International Agricultural Trade Research Consortium.
    22. repec:wbk:wbpubs:2693 is not listed on IDEAS
    23. David Roodman, 2009. "A Note on the Theme of Too Many Instruments," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 71(1), pages 135-158, February.
    24. Ritva Reinikka & Jakob Svensson, 2005. "Fighting Corruption to Improve Schooling: Evidence from a Newspaper Campaign in Uganda," Journal of the European Economic Association, MIT Press, vol. 3(2-3), pages 259-267, 04/05.
    25. Natalie Carvalho & Slawa Rokicki, 2019. "The Impact of India’s Janani Suraksha Yojana Conditional Cash Transfer Programme: A Replication Study," Journal of Development Studies, Taylor & Francis Journals, vol. 55(5), pages 989-1006, May.
    26. Nava Ashraf & Xavier Giné & Dean Karlan, 2009. "Finding Missing Markets (and a Disturbing Epilogue): Evidence from an Export Crop Adoption and Marketing Intervention in Kenya," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 91(4), pages 973-990.
    27. Rajeev K. Goel & Ummad Mazhar, 2015. "A Replication of “Corruption and Elections," Public Finance Review, , vol. 43(2), pages 143-154, March.
    28. Robert Jensen & Emily Oster, 2009. "The Power of TV: Cable Television and Women's Status in India," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 124(3), pages 1057-1094.
    29. Jeffrey R Kling & Jeffrey B Liebman & Lawrence F Katz, 2007. "Experimental Analysis of Neighborhood Effects," Econometrica, Econometric Society, vol. 75(1), pages 83-119, January.
    30. Jesse Rothstein, 2017. "Measuring the Impacts of Teachers: Comment," American Economic Review, American Economic Association, vol. 107(6), pages 1656-1684, June.
    31. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-620, September.
    32. Patrick Button, 2015. "A Replication of 'Do Voters Affect or Elect Policies? Evidence from the U.S. House' (The Quarterly Journal of Economics, 2004)," Working Papers 1518, Tulane University, Department of Economics.
    33. repec:wbk:wbpubs:25030 is not listed on IDEAS
    34. Sandip Sukhtankar, 2017. "Replications in Development Economics," American Economic Review, American Economic Association, vol. 107(5), pages 32-36, May.
    35. Helland Eric & Tabarrok Alexander, 2004. "Using Placebo Laws to Test "More Guns, Less Crime"," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 4(1), pages 1-9, January.
    36. Kilic, Talip & Zezza, Alberto & Carletto, Calogero & Savastano, Sara, 2017. "Missing(ness) in Action: Selectivity Bias in GPS-Based Land Area Measurements," World Development, Elsevier, vol. 92(C), pages 143-157.
    37. Guido Imbens & Karthik Kalyanaraman, 2012. "Optimal Bandwidth Choice for the Regression Discontinuity Estimator," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 79(3), pages 933-959.
    38. Daron Acemoglu & Simon Johnson & James A. Robinson & Pierre Yared, 2008. "Income and Democracy," American Economic Review, American Economic Association, vol. 98(3), pages 808-842, June.
    39. Marco Caliendo & Sabine Kopeinig, 2008. "Some Practical Guidance For The Implementation Of Propensity Score Matching," Journal of Economic Surveys, Wiley Blackwell, vol. 22(1), pages 31-72, February.
    40. Lall, Ranjit, 2016. "How Multiple Imputation Makes a Difference," Political Analysis, Cambridge University Press, vol. 24(4), pages 414-433.
    41. Edward M. Whitney & Drew B. Cameron & Paul C. Winters, 2019. "Heterogeneous Effects of Urban Land Titling: A Replication of ‘Property Rights for the Poor’," Journal of Development Studies, Taylor & Francis Journals, vol. 55(5), pages 1030-1033, May.
    42. Ashraf, Nava & Giné, Xavier & Karlan, Dean S., . "AJAE appendix for “Finding Missing Markets (and a Disturbing Epilogue): Evidence from an Export Crop Adoption and Marketing Intervention in Kenya”," American Journal of Agricultural Economics APPENDICES, Agricultural and Applied Economics Association, vol. 91(4), pages 1-9.
    43. Matteo Cervellati & Florian Jung & Uwe Sunde & Thomas Vischer, 2014. "Income and Democracy: Comment," American Economic Review, American Economic Association, vol. 104(2), pages 707-719, February.
    44. Sebastian Calonico & Matias D. Cattaneo & Rocío Titiunik, 2015. "Optimal Data-Driven Regression Discontinuity Plots," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(512), pages 1753-1769, December.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Wood, Benjamin Douglas Kuflick & Vasquez, Maria, 2018. "Microplots and food security: Encouraging replication studies of policy relevant research," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy, vol. 12, pages 1-12.
    2. Reed, W. Robert, 2019. "Takeaways from the special issue on The Practice of Replication," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy, vol. 13, pages 1-11.
    3. Owen, P. Dorian, 2018. "Replication to assess statistical adequacy," Economics - The Open-Access, Open-Assessment E-Journal (2007-2020), Kiel Institute for the World Economy, vol. 12, pages 1-16.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Sant’Anna, Pedro H.C. & Zhao, Jun, 2020. "Doubly robust difference-in-differences estimators," Journal of Econometrics, Elsevier, vol. 219(1), pages 101-122.
    2. Ari Hyytinen & Jaakko Meriläinen & Tuukka Saarimaa & Otto Toivanen & Janne Tukiainen, 2018. "When does regression discontinuity design work? Evidence from random election outcomes," Quantitative Economics, Econometric Society, vol. 9(2), pages 1019-1051, July.
    3. Rothstein, Jesse & Von Wachter, Till, 2016. "Social Experiments in the Labor Market," Department of Economics, Working Paper Series qt7957p9g6, Department of Economics, Institute for Business and Economic Research, UC Berkeley.
    4. Dettmann, E. & Becker, C. & Schmeißer, C., 2011. "Distance functions for matching in small samples," Computational Statistics & Data Analysis, Elsevier, vol. 55(5), pages 1942-1960, May.
    5. Caliendo, Marco & Mahlstedt, Robert & Mitnik, Oscar A., 2017. "Unobservable, but unimportant? The relevance of usually unobserved variables for the evaluation of labor market policies," Labour Economics, Elsevier, vol. 46(C), pages 14-25.
    6. Jens Ruhose & Stephan L. Thomsen & Insa Weilage, 2018. "The Wider Benefits of Adult Learning: Work-Related Training and Social Capital," CESifo Working Paper Series 7268, CESifo.
    7. Brendon McConnell & Marcos Vera‐Hernández, 2025. "Going beyond simple sample size calculations: a practitioner's guide," Fiscal Studies, John Wiley & Sons, vol. 46(3), pages 323-348, September.
    8. Ruhose, Jens & Thomsen, Stephan, 2017. "Non-Monetary Benefits of Continuous Training," VfS Annual Conference 2017 (Vienna): Alternative Structures for Money and Banking 168169, Verein für Socialpolitik / German Economic Association.
    9. Jens Ruhose & Stephan L. Thomsen & Insa Weilage, 2018. "The Wider Benefits of Adult Learning: Work-Related Training and Social Capital," SOEPpapers on Multidisciplinary Panel Data Research 1004, DIW Berlin, The German Socio-Economic Panel (SOEP).
    10. Doris Läpple & Thia Hennessy, 2015. "Assessing the Impact of Financial Incentives in Extension Programmes: Evidence From Ireland," Journal of Agricultural Economics, Wiley Blackwell, vol. 66(3), pages 781-795, September.
    11. Arouna, Aminou & Michler, Jeffrey D. & Lokossou, Jourdain C., 2021. "Contract farming and rural transformation: Evidence from a field experiment in Benin," Journal of Development Economics, Elsevier, vol. 151(C).
    12. Brendon McConnell & Marcos Vera-Hernandez, 2015. "Going beyond simple sample size calculations: a practitioner's guide," IFS Working Papers W15/17, Institute for Fiscal Studies.
    13. Justus Ochieng & Beatrice Knerr & George Owuor & Emily Ouma, 2020. "Food crops commercialization and household livelihoods: Evidence from rural regions in Central Africa," Agribusiness, John Wiley & Sons, Ltd., vol. 36(2), pages 318-338, April.
    14. Jacopo, Bonan & Stefano, Pareglio & Valentina, Rotondi, 2015. "Extension Services, Production and Welfare: Evidence from a Field Experiment in Ethiopia," Working Papers 312, University of Milano-Bicocca, Department of Economics, revised 30 Oct 2015.
    15. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    16. Luis Aranda, 2013. "Doubling Up: A Gift or a Shame? Multigenerational Households and Parental Depression of Older Europeans," Working Papers 2013:29, Department of Economics, University of Venice "Ca' Foscari", revised 2013.
    17. Doerr, Leo M & Leppert, Elias B & Maennig, Wolfgang, 2025. "Olympic Games and democracy," Economic Analysis and Policy, Elsevier, vol. 87(C), pages 1073-1091.
    18. Ruhose, Jens & Thomsen, Stephan L. & Weilage, Insa, 2019. "The benefits of adult learning: Work-related training, social capital, and earnings," Economics of Education Review, Elsevier, vol. 72(C), pages 166-186.
    19. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    20. Sokbae (Simon) Lee & Yoon-Jae Whang, 2009. "Nonparametric tests of conditional treatment effects," CeMMAP working papers CWP36/09, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;
    ;

    JEL classification:

    • C10 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - General
    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • A20 - General Economics and Teaching - - Economic Education and Teaching of Economics - - - General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:ifwedp:201777. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/iwkiede.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.