IDEAS home Printed from https://ideas.repec.org/a/zbw/ifweej/201853.html
   My bibliography  Save this article

Which tests not witch hunts: A diagnostic approach for conducting replication research

Author

Listed:
  • Brown, Annette N.
  • Wood, Benjamin Douglas Kuflick

Abstract

Replication research can be used to explore original study results that researchers consider questionable, but it should also be a tool for reinforcing the credibility of results that are important to policies and programs. The challenge is to design a replication plan open to both supporting the original findings and uncovering potential problems. The purpose of this paper is to provide replication researchers with an objective list of checks or tests to consider when planning a replication study. The authors present tips for diagnostic replication exercises in four groups: validity of assumptions, data transformations, estimation methods, and heterogeneous impacts. For each group, the authors present an introduction to the issues, a list of replication tests and checks, some examples of how these checks are employed in replication studies of development impact evaluations, and a set of resources that provide statistical and econometric details. The authors also provide a list of don'ts for how to conduct and report replication research.

Suggested Citation

  • Brown, Annette N. & Wood, Benjamin Douglas Kuflick, 2018. "Which tests not witch hunts: A diagnostic approach for conducting replication research," Economics - The Open-Access, Open-Assessment E-Journal, Kiel Institute for the World Economy (IfW), vol. 12, pages 1-26.
  • Handle: RePEc:zbw:ifweej:201853
    DOI: 10.5018/economics-ejournal.ja.2018-53
    as

    Download full text from publisher

    File URL: http://dx.doi.org/10.5018/economics-ejournal.ja.2018-53
    Download Restriction: no

    File URL: https://www.econstor.eu/bitstream/10419/181452/1/1029195676.pdf
    Download Restriction: no

    File URL: https://libkey.io/10.5018/economics-ejournal.ja.2018-53?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. David McKenzie, 2017. "How Effective Are Active Labor Market Policies in Developing Countries? A Critical Review of Recent Evidence," World Bank Research Observer, World Bank Group, vol. 32(2), pages 127-154.
    2. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    3. Cummins, Joseph R., 2017. "Heterogeneous treatment effects in the low track: Revisiting the Kenyan primary school experiment," Economics of Education Review, Elsevier, vol. 56(C), pages 40-51.
    4. Nava Ashraf & Xavier Giné & Dean Karlan, 2009. "Finding Missing Markets (and a Disturbing Epilogue): Evidence from an Export Crop Adoption and Marketing Intervention in Kenya," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 91(4), pages 973-990.
    5. Daron Acemoglu & Simon Johnson & James A. Robinson & Pierre Yared, 2008. "Income and Democracy," American Economic Review, American Economic Association, vol. 98(3), pages 808-842, June.
    6. Robert Jensen & Emily Oster, 2009. "The Power of TV: Cable Television and Women's Status in India," The Quarterly Journal of Economics, Oxford University Press, vol. 124(3), pages 1057-1094.
    7. Paul J. Gertler & Sebastian Martinez & Patrick Premand & Laura B. Rawlings & Christel M. J. Vermeersch, 2016. "Impact Evaluation in Practice, Second Edition," World Bank Publications, The World Bank, number 25030.
    8. Kilic, Talip & Zezza, Alberto & Carletto, Calogero & Savastano, Sara, 2017. "Missing(ness) in Action: Selectivity Bias in GPS-Based Land Area Measurements," World Development, Elsevier, vol. 92(C), pages 143-157.
    9. Benjamin D K Wood & Rui Müller & Annette N Brown, 2018. "Push button replication: Is impact evaluation evidence for international development verifiable?," PLOS ONE, Public Library of Science, vol. 13(12), pages 1-15, December.
    10. Marco Caliendo & Sabine Kopeinig, 2008. "Some Practical Guidance For The Implementation Of Propensity Score Matching," Journal of Economic Surveys, Wiley Blackwell, vol. 22(1), pages 31-72, February.
    11. Jeffrey R Kling & Jeffrey B Liebman & Lawrence F Katz, 2007. "Experimental Analysis of Neighborhood Effects," Econometrica, Econometric Society, vol. 75(1), pages 83-119, January.
    12. David Roodman, 2009. "A Note on the Theme of Too Many Instruments," Oxford Bulletin of Economics and Statistics, Department of Economics, University of Oxford, vol. 71(1), pages 135-158, February.
    13. Jesse Rothstein, 2017. "Measuring the Impacts of Teachers: Comment," American Economic Review, American Economic Association, vol. 107(6), pages 1656-1684, June.
    14. Anderson, Michael L., 2008. "Multiple Inference and Gender Differences in the Effects of Early Intervention: A Reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects," Journal of the American Statistical Association, American Statistical Association, vol. 103(484), pages 1481-1495.
    15. Pradyot Ranjan Jena & Bezawit Beyene Chichaibelu & Till Stellmacher & Ulrike Grote, 2012. "The impact of coffee certification on small-scale producers’ livelihoods: a case study from the Jimma Zone, Ethiopia," Agricultural Economics, International Association of Agricultural Economists, vol. 43(4), pages 429-440, July.
    16. A. Smith, Jeffrey & E. Todd, Petra, 2005. "Does matching overcome LaLonde's critique of nonexperimental estimators?," Journal of Econometrics, Elsevier, vol. 125(1-2), pages 305-353.
    17. Christian Zimmermann, 2015. "On the Need for a Replication Journal," Working Papers 2015-16, Federal Reserve Bank of St. Louis.
    18. Jesse Rothstein, 2010. "Teacher Quality in Educational Production: Tracking, Decay, and Student Achievement," The Quarterly Journal of Economics, Oxford University Press, vol. 125(1), pages 175-214.
    19. Matias D. Cattaneo & Sebastian Galiani & Paul J. Gertler & Sebastian Martinez & Rocio Titiunik, 2009. "Housing, Health, and Happiness," American Economic Journal: Economic Policy, American Economic Association, vol. 1(1), pages 75-105, February.
    20. Natalie Carvalho & Slawa Rokicki, 2019. "The Impact of India’s Janani Suraksha Yojana Conditional Cash Transfer Programme: A Replication Study," Journal of Development Studies, Taylor & Francis Journals, vol. 55(5), pages 989-1006, May.
    21. Stefan Dercon & Daniel O. Gilligan & John Hoddinott & Tassew Woldehanna, 2009. "The Impact of Agricultural Extension and Roads on Poverty and Consumption Growth in Fifteen Ethiopian Villages," American Journal of Agricultural Economics, Agricultural and Applied Economics Association, vol. 91(4), pages 1007-1021.
    22. Galiani, Sebastian & Schargrodsky, Ernesto, 2010. "Property rights for the poor: Effects of land titling," Journal of Public Economics, Elsevier, vol. 94(9-10), pages 700-729, October.
    23. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    24. Raj Chetty & John N. Friedman & Jonah E. Rockoff, 2017. "Measuring the Impacts of Teachers: Reply," American Economic Review, American Economic Association, vol. 107(6), pages 1685-1717, June.
    25. Guido Imbens & Karthik Kalyanaraman, 2012. "Optimal Bandwidth Choice for the Regression Discontinuity Estimator," Review of Economic Studies, Oxford University Press, vol. 79(3), pages 933-959.
    26. Lall, Ranjit, 2016. "How Multiple Imputation Makes a Difference," Political Analysis, Cambridge University Press, vol. 24(4), pages 414-433.
    27. LaLonde, Robert J, 1986. "Evaluating the Econometric Evaluations of Training Programs with Experimental Data," American Economic Review, American Economic Association, vol. 76(4), pages 604-620, September.
    28. Imai, Kosuke & Strauss, Aaron, 2011. "Estimation of Heterogeneous Treatment Effects from Randomized Experiments, with Application to the Optimal Planning of the Get-Out-the-Vote Campaign," Political Analysis, Cambridge University Press, vol. 19(1), pages 1-19, January.
    29. Sebastian Calonico & Matias D. Cattaneo & Rocío Titiunik, 2015. "Optimal Data-Driven Regression Discontinuity Plots," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 110(512), pages 1753-1769, December.
    30. David S. Lee & Enrico Moretti & Matthew J. Butler, 2004. "Do Voters Affect or Elect Policies? Evidence from the U. S. House," The Quarterly Journal of Economics, Oxford University Press, vol. 119(3), pages 807-859.
    31. Patrick Button, 2015. "A Replication of 'Do Voters Affect or Elect Policies? Evidence from the U.S. House' (The Quarterly Journal of Economics, 2004)," Working Papers 1518, Tulane University, Department of Economics.
    32. Edward M. Whitney & Drew B. Cameron & Paul C. Winters, 2019. "Heterogeneous Effects of Urban Land Titling: A Replication of ‘Property Rights for the Poor’," Journal of Development Studies, Taylor & Francis Journals, vol. 55(5), pages 1030-1033, May.
    33. Shahidur R. Khandker & Gayatri B. Koolwal & Hussain A. Samad, . "Handbook on Impact Evaluation : Quantitative Methods and Practices," World Bank Publications, The World Bank, number 2693.
    34. Rajeev K. Goel & Ummad Mazhar, 2015. "A Replication of “Corruption and Elections," Public Finance Review, , vol. 43(2), pages 143-154, March.
    35. Matteo Cervellati & Florian Jung & Uwe Sunde & Thomas Vischer, 2014. "Income and Democracy: Comment," American Economic Review, American Economic Association, vol. 104(2), pages 707-719, February.
    36. Sebastian Galiani & Paul Gertler & Mauricio Romero, 2017. "Incentives for Replication in Economics," NBER Working Papers 23576, National Bureau of Economic Research, Inc.
    37. Annette N. Brown & Drew B. Cameron & Benjamin D. K. Wood, 2014. "Quality evidence for policymaking: I'll believe it when I see the replication," Journal of Development Effectiveness, Taylor & Francis Journals, vol. 6(3), pages 215-235, September.
    38. Ritva Reinikka & Jakob Svensson, 2005. "Fighting Corruption to Improve Schooling: Evidence from a Newspaper Campaign in Uganda," Journal of the European Economic Association, MIT Press, vol. 3(2-3), pages 259-267, 04/05.
    39. Sandip Sukhtankar, 2017. "Replications in Development Economics," American Economic Review, American Economic Association, vol. 107(5), pages 32-36, May.
    40. Ashraf, Nava & Giné, Xavier & Karlan, Dean S., 2009. "AJAE appendix for “Finding Missing Markets (and a Disturbing Epilogue): Evidence from an Export Crop Adoption and Marketing Intervention in Kenya”," American Journal of Agricultural Economics APPENDICES, Agricultural and Applied Economics Association, vol. 91(4), pages 1-9, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Wood, Benjamin Douglas Kuflick & Vasquez, Maria, 2018. "Microplots and food security: Encouraging replication studies of policy relevant research," Economics - The Open-Access, Open-Assessment E-Journal, Kiel Institute for the World Economy (IfW), vol. 12, pages 1-12.
    2. Reed, W. Robert, 2019. "Takeaways from the special issue on The Practice of Replication," Economics - The Open-Access, Open-Assessment E-Journal, Kiel Institute for the World Economy (IfW), vol. 13, pages 1-11.
    3. Owen, P. Dorian, 2018. "Replication to assess statistical adequacy," Economics - The Open-Access, Open-Assessment E-Journal, Kiel Institute for the World Economy (IfW), vol. 12, pages 1-16.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    2. Ari Hyytinen & Jaakko Meriläinen & Tuukka Saarimaa & Otto Toivanen & Janne Tukiainen, 2018. "When does regression discontinuity design work? Evidence from random election outcomes," Quantitative Economics, Econometric Society, vol. 9(2), pages 1019-1051, July.
    3. Dettmann, E. & Becker, C. & Schmeißer, C., 2011. "Distance functions for matching in small samples," Computational Statistics & Data Analysis, Elsevier, vol. 55(5), pages 1942-1960, May.
    4. Ashimwe, Olive, 2016. "An Economic Analysis Of Impact Of Weather Index-Based Crop Insurance On Household Income In Huye District Of Rwanda," Research Theses 265675, Collaborative Masters Program in Agricultural and Applied Economics.
    5. Del Prete, Davide & Ghins, Léopold & Magrini, Emiliano & Pauw, Karl, 2019. "Land consolidation, specialization and household diets: Evidence from Rwanda," Food Policy, Elsevier, vol. 83(C), pages 139-149.
    6. Caliendo, Marco & Mahlstedt, Robert & Mitnik, Oscar A., 2017. "Unobservable, but unimportant? The relevance of usually unobserved variables for the evaluation of labor market policies," Labour Economics, Elsevier, vol. 46(C), pages 14-25.
    7. Ruhose, Jens & Thomsen, Stephan L. & Weilage, Insa, 2018. "The Wider Benefits of Adult Learning: Work-Related Training and Social Capital," IZA Discussion Papers 11854, Institute of Labor Economics (IZA).
    8. Lajos Baráth & Imre Fertő & Štefan Bojnec, 2020. "The Effect of Investment, LFA and Agri‐environmental Subsidies on the Components of Total Factor Productivity: The Case of Slovenian Farms," Journal of Agricultural Economics, Wiley Blackwell, vol. 71(3), pages 853-876, September.
    9. Ogutu, Sylvester Ochieng & Okello, Julius Juma & Otieno, David Jakinda, 2014. "Impact of Information and Communication Technology-Based Market Information Services on Smallholder Farm Input Use and Productivity: The Case of Kenya," World Development, Elsevier, vol. 64(C), pages 311-321.
    10. Sokbae Lee & Yoon-Jae Whang, 2009. "Nonparametric Tests of Conditional Treatment Effects," Cowles Foundation Discussion Papers 1740, Cowles Foundation for Research in Economics, Yale University.
    11. Bac Ho & Teruaki Nanseki & Yosuke Chomei, 2019. "Profit efficiency of tea farmers: case study of safe and conventional farms in Northern Vietnam," Environment, Development and Sustainability: A Multidisciplinary Approach to the Theory and Practice of Sustainable Development, Springer, vol. 21(4), pages 1695-1713, August.
    12. Rothstein, Jesse & von Wachter, Till, 2016. "Social Experiments in the Labor Market," Institute for Research on Labor and Employment, Working Paper Series qt6605k20b, Institute of Industrial Relations, UC Berkeley.
    13. Ruhose, Jens & Thomsen, Stephan, 2017. "Non-Monetary Benefits of Continuous Training," VfS Annual Conference 2017 (Vienna): Alternative Structures for Money and Banking 168169, Verein für Socialpolitik / German Economic Association.
    14. Aminou Arouna & Jeffrey D. Michler & Jourdain C. Lokossou, 2019. "Contract Farming and Rural Transformation: Evidence from a Field Experiment in Benin," NBER Working Papers 25665, National Bureau of Economic Research, Inc.
    15. Jens Ruhose & Stephan L. Thomsen & Insa Weilage, 2018. "The Wider Benefits of Adult Learning: Work-Related Training and Social Capital," SOEPpapers on Multidisciplinary Panel Data Research 1004, DIW Berlin, The German Socio-Economic Panel (SOEP).
    16. Jens Ruhose & Stephan L. Thomsen & Insa Weilage, 2018. "The Wider Benefits of Adult Learning: Work-Related Training and Social Capital," CESifo Working Paper Series 7268, CESifo.
    17. Justus Ochieng & Beatrice Knerr & George Owuor & Emily Ouma, 2020. "Food crops commercialization and household livelihoods: Evidence from rural regions in Central Africa," Agribusiness, John Wiley & Sons, Ltd., vol. 36(2), pages 318-338, April.
    18. Rothstein, Jesse & von Wachter, Till, 2016. "Social Experiments in the Labor Market," Department of Economics, Working Paper Series qt6605k20b, Department of Economics, Institute for Business and Economic Research, UC Berkeley.
    19. Jacopo, Bonan & Stefano, Pareglio & Valentina, Rotondi, 2015. "Extension Services, Production and Welfare: Evidence from a Field Experiment in Ethiopia," Working Papers 312, University of Milano-Bicocca, Department of Economics, revised 30 Oct 2015.
    20. Rothstein, J & von Wachter, T, 2016. "Social Experiments in the Labor Market," Department of Economics, Working Paper Series qt7957p9g6, Department of Economics, Institute for Business and Economic Research, UC Berkeley.

    More about this item

    Keywords

    replication; validation; data transformation; estimation methods; reanalysis; theory of change; assumptions;
    All these keywords.

    JEL classification:

    • C10 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - General
    • B41 - Schools of Economic Thought and Methodology - - Economic Methodology - - - Economic Methodology
    • A20 - General Economics and Teaching - - Economic Education and Teaching of Economics - - - General

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zbw:ifweej:201853. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/iwkiede.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: ZBW - Leibniz Information Centre for Economics (email available below). General contact details of provider: https://edirc.repec.org/data/iwkiede.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.