IDEAS home Printed from https://ideas.repec.org/a/zbw/ifweej/201853.html
   My bibliography  Save this article

Which tests not witch hunts: A diagnostic approach for conducting replication research

Author

Listed:
  • Brown, Annette N.
  • Wood, Benjamin Douglas Kuflick

Abstract

Replication research can be used to explore original study results that researchers consider questionable, but it should also be a tool for reinforcing the credibility of results that are important to policies and programs. The challenge is to design a replication plan open to both supporting the original findings and uncovering potential problems. The purpose of this paper is to provide replication researchers with an objective list of checks or tests to consider when planning a replication study. The authors present tips for diagnostic replication exercises in four groups: validity of assumptions, data transformations, estimation methods, and heterogeneous impacts. For each group, the authors present an introduction to the issues, a list of replication tests and checks, some examples of how these checks are employed in replication studies of development impact evaluations, and a set of resources that provide statistical and econometric details. The authors also provide a list of don'ts for how to conduct and report replication research.

Suggested Citation

  • Brown, Annette N. & Wood, Benjamin Douglas Kuflick, 2018. "Which tests not witch hunts: A diagnostic approach for conducting replication research," Economics - The Open-Access, Open-Assessment E-Journal, Kiel Institute for the World Economy (IfW), vol. 12, pages 1-26.
  • Handle: RePEc:zbw:ifweej:201853
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.5018/economics-ejournal.ja.2018-53
    Download Restriction: no

    File URL: https://www.econstor.eu/bitstream/10419/181452/1/1029195676.pdf
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    as
    1. David McKenzie, 2017. "How Effective Are Active Labor Market Policies in Developing Countries? A Critical Review of Recent Evidence," World Bank Research Observer, World Bank Group, vol. 32(2), pages 127-154.
    2. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-620, September.
    3. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    4. Cummins, Joseph R., 2017. "Heterogeneous treatment effects in the low track: Revisiting the Kenyan primary school experiment," Economics of Education Review, Elsevier, vol. 56(C), pages 40-51.
    5. Imai, Kosuke & Strauss, Aaron, 2011. "Estimation of Heterogeneous Treatment Effects from Randomized Experiments, with Application to the Optimal Planning of the Get-Out-the-Vote Campaign," Political Analysis, Cambridge University Press, vol. 19(1), pages 1-19, January.
    6. Jesse Rothstein, 2010. "Teacher Quality in Educational Production: Tracking, Decay, and Student Achievement," The Quarterly Journal of Economics, Oxford University Press, vol. 125(1), pages 175-214.
    7. Stefan Dercon & Daniel O. Gilligan & John Hoddinott & Tassew Woldehanna, 2009. "The Impact of Agricultural Extension and Roads on Poverty and Consumption Growth in Fifteen Ethiopian Villages," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 91(4), pages 1007-1021.
    8. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    9. Nava Ashraf & Xavier Giné & Dean Karlan, 2009. "Finding Missing Markets (and a Disturbing Epilogue): Evidence from an Export Crop Adoption and Marketing Intervention in Kenya," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 91(4), pages 973-990.
    10. Galiani, Sebastian & Schargrodsky, Ernesto, 2010. "Property rights for the poor: Effects of land titling," Journal of Public Economics, Elsevier, vol. 94(9-10), pages 700-729, October.
    11. Matias D. Cattaneo & Sebastian Galiani & Paul J. Gertler & Sebastian Martinez & Rocio Titiunik, 2009. "Housing, Health, and Happiness," American Economic Journal: Economic Policy, American Economic Association, vol. 1(1), pages 75-105, February.
    12. Daron Acemoglu & Simon Johnson & James A. Robinson & Pierre Yared, 2008. "Income and Democracy," American Economic Review, American Economic Association, vol. 98(3), pages 808-842, June.
    13. Robert Jensen & Emily Oster, 2009. "The Power of TV: Cable Television and Women's Status in India," The Quarterly Journal of Economics, Oxford University Press, vol. 124(3), pages 1057-1094.
    14. Paul J. Gertler & Sebastian Martinez & Patrick Premand & Laura B. Rawlings & Christel M. J. Vermeersch, 2016. "Impact Evaluation in Practice, Second Edition," World Bank Publications, The World Bank, number 25030.
    15. Kilic, Talip & Zezza, Alberto & Carletto, Calogero & Savastano, Sara, 2017. "Missing(ness) in Action: Selectivity Bias in GPS-Based Land Area Measurements," World Development, Elsevier, vol. 92(C), pages 143-157.
    16. Marco Caliendo & Sabine Kopeinig, 2008. "Some Practical Guidance For The Implementation Of Propensity Score Matching," Journal of Economic Surveys, Wiley Blackwell, vol. 22(1), pages 31-72, February.
    17. Sebastian Calonico & Matias D. Cattaneo & Rocío Titiunik, 2015. "Optimal Data-Driven Regression Discontinuity Plots," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(512), pages 1753-1769, December.
    18. David S. Lee & Enrico Moretti & Matthew J. Butler, 2004. "Do Voters Affect or Elect Policies? Evidence from the U. S. House," The Quarterly Journal of Economics, Oxford University Press, vol. 119(3), pages 807-859.
    19. Jeffrey R Kling & Jeffrey B Liebman & Lawrence F Katz, 2007. "Experimental Analysis of Neighborhood Effects," Econometrica, Econometric Society, vol. 75(1), pages 83-119, January.
    20. Guido Imbens & Karthik Kalyanaraman, 2012. "Optimal Bandwidth Choice for the Regression Discontinuity Estimator," Review of Economic Studies, Oxford University Press, vol. 79(3), pages 933-959.
    21. David Roodman, 2009. "A Note on the Theme of Too Many Instruments," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 71(1), pages 135-158, February.
    22. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    23. repec:aea:aecrev:v:107:y:2017:i:6:p:1656-84 is not listed on IDEAS
    24. Patrick Button, 2015. "A Replication of 'Do Voters Affect or Elect Policies? Evidence from the U.S. House' (The Quarterly Journal of Economics, 2004)," Working Papers 1518, Tulane University, Department of Economics.
    25. Anderson, Michael L., 2008. "Multiple Inference and Gender Differences in the Effects of Early Intervention: A Reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects," Journal of the American Statistical Association, American Statistical Association, vol. 103(484), pages 1481-1495.
    26. repec:taf:jdevst:v:55:y:2019:i:5:p:1030-1033 is not listed on IDEAS
    27. Pradyot Ranjan Jena & Bezawit Beyene Chichaibelu & Till Stellmacher & Ulrike Grote, 2012. "The impact of coffee certification on small-scale producers’ livelihoods: a case study from the Jimma Zone, Ethiopia," Agricultural Economics, International Association of Agricultural Economists, vol. 43(4), pages 429-440, July.
    28. Zimmermann, Christian, 2015. "On the Need for a Replication Journal," Working Papers 2015-16, Federal Reserve Bank of St. Louis.
    29. Shahidur R. Khandker & Gayatri B. Koolwal & Hussain A. Samad, . "Handbook on Impact Evaluation : Quantitative Methods and Practices," World Bank Publications, The World Bank, number 2693.
    30. repec:taf:jdevst:v:55:y:2019:i:5:p:989-1006 is not listed on IDEAS
    31. Rajeev K. Goel & Ummad Mazhar, 2015. "A Replication of “Corruption and Elections," Public Finance Review, , vol. 43(2), pages 143-154, March.
    32. Matteo Cervellati & Florian Jung & Uwe Sunde & Thomas Vischer, 2014. "Income and Democracy: Comment," American Economic Review, American Economic Association, vol. 104(2), pages 707-719, February.
    33. repec:aea:aecrev:v:107:y:2017:i:6:p:1685-1717 is not listed on IDEAS
    34. Sebastian Galiani & Paul Gertler & Mauricio Romero, 2017. "Incentives for Replication in Economics," NBER Working Papers 23576, National Bureau of Economic Research, Inc.
    35. Annette N. Brown & Drew B. Cameron & Benjamin D. K. Wood, 2014. "Quality evidence for policymaking: I'll believe it when I see the replication," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(3), pages 215-235, September.
    36. Ritva Reinikka & Jakob Svensson, 2005. "Fighting Corruption to Improve Schooling: Evidence from a Newspaper Campaign in Uganda," Journal of the European Economic Association, MIT Press, vol. 3(2-3), pages 259-267, 04/05.
    37. repec:aea:aecrev:v:107:y:2017:i:5:p:32-36 is not listed on IDEAS
    38. Lall, Ranjit, 2016. "How Multiple Imputation Makes a Difference," Political Analysis, Cambridge University Press, vol. 24(4), pages 414-433.
    39. Ashraf, Nava & Giné, Xavier & Karlan, Dean S., 2009. "AJAE appendix for “Finding Missing Markets (and a Disturbing Epilogue): Evidence from an Export Crop Adoption and Marketing Intervention in Kenya”," American Journal of Agricultural Economics APPENDICES, Agricultural and Applied Economics Association, vol. 91(4), pages 1-9, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. repec:zbw:ifweej:201850 is not listed on IDEAS
    2. Reed, W. Robert, 2019. "Takeaways from the special issue on The Practice of Replication," Economics - The Open-Access, Open-Assessment E-Journal, Kiel Institute for the World Economy (IfW), vol. 13, pages 1-11.
    3. repec:zbw:ifweej:201860 is not listed on IDEAS
    4. Wood, Benjamin Douglas Kuflick & Vasquez, Maria, 2018. "Microplots and food security: Encouraging replication studies of policy relevant research," Economics - The Open-Access, Open-Assessment E-Journal, Kiel Institute for the World Economy (IfW), vol. 12, pages 1-12.
    5. Owen, P. Dorian, 2018. "Replication to assess statistical adequacy," Economics - The Open-Access, Open-Assessment E-Journal, Kiel Institute for the World Economy (IfW), vol. 12, pages 1-16.

    More about this item

    Keywords

    replication; validation; data transformation; estimation methods; reanalysis; theory of change; assumptions;

    JEL classification:

    • C10 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - General
    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • A20 - General Economics and Teaching - - Economic Education and Teaching of Economics - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:ifweej:201853. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (ZBW - Leibniz Information Centre for Economics). General contact details of provider: http://edirc.repec.org/data/iwkiede.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.