IDEAS home Printed from https://ideas.repec.org/
MyIDEAS: Login to save this paper or follow this series

Testing for Heterogeneous Treatment Effects in Experimental Data: False Discovery Risks and Correction Procedures

  • Fink, Günther
  • McConnell, Margaret
  • Vollmer, Sebastian
Registered author(s):

    We review the statistical models applied to test for heterogeneous treatment effects in the recent empirical literature, with a particular focus on data from randomized field experiments. We show that testing for heterogeneous treatment effects is highly common, and likely to result in a large number of false discoveries when conventional standard errors are applied. We demonstrate that applying correction procedures developed in the statistics literature can fully address this issue, and discuss the implications of multiple testing adjustments for power calculations and experimental design.

    If you experience problems downloading a file, check if you have the proper application to view it first. In case of further problems read the IDEAS help page. Note that these files are not on the IDEAS site. Please be patient as the files may be large.

    File URL: http://diskussionspapiere.wiwi.uni-hannover.de/pdf_bib/dp-477.pdf
    Download Restriction: no

    Paper provided by Leibniz Universität Hannover, Wirtschaftswissenschaftliche Fakultät in its series Hannover Economic Papers (HEP) with number dp-477.

    as
    in new window

    Length: 26 pages
    Date of creation: Jul 2011
    Date of revision:
    Handle: RePEc:han:dpaper:dp-477
    Contact details of provider: Postal:
    Koenigsworther Platz 1, D-30167 Hannover

    Phone: (0511) 762-5350
    Fax: (0511) 762-5665
    Web page: http://www.wiwi.uni-hannover.de

    More information through EDIRC

    References listed on IDEAS
    Please report citation or reference errors to , or , if you are the registered author of the cited work, log in to your RePEc Author Service profile, click on "citations" and make appropriate adjustments.:

    as in new window
    1. de Mel, Suresh & McKenzie, David & Woodruff, Christopher, 2007. "Returns to Capital in Microenterprises: Evidence from a Field Experiment," IZA Discussion Papers 2934, Institute for the Study of Labor (IZA).
    2. Armin Falk, 2007. "Gift Exchange in the Field," Econometrica, Econometric Society, vol. 75(5), pages 1501-1511, 09.
    3. David Card & Stefano DellaVigna & Ulrike Malmendier, 2011. "The Role of Theory in Field Experiments," NBER Working Papers 17047, National Bureau of Economic Research, Inc.
    4. Marianne P. Bitler & Jonah B. Gelbach & Hilary W. Hoynes, 2003. "What Mean Impacts Miss: Distributional Effects of Welfare Reform Experiments," Working Papers 109, RAND Corporation Publications Department.
    5. Rebecca L. Thornton, 2008. "The Demand for, and Impact of, Learning HIV Status," American Economic Review, American Economic Association, vol. 98(5), pages 1829-63, December.
    6. Uri Gneezy & John A. List, 2006. "Putting Behavioral Economics to Work: Testing for Gift Exchange in Labor Markets Using Field Experiments," NBER Working Papers 12063, National Bureau of Economic Research, Inc.
    7. Joshua Angrist & Victor Lavy, 2009. "The Effects of High Stakes High School Achievement Awards: Evidence from a Randomized Trial," American Economic Review, American Economic Association, vol. 99(4), pages 1384-1414, September.
    8. Kristie M. Engemann & Howard J. Wall, 2009. "A journal ranking for the ambitious economist," Review, Federal Reserve Bank of St. Louis, issue May, pages 127-140.
    9. Kremer, Michael Robert & Miguel, Edward A. & Thorton, Rebecca L, 2004. "Incentives to Learn," Center for International and Development Economics Research, Working Paper Series qt9kc4p47q, Center for International and Development Economics Research, Institute for Business and Economic Research, UC Berkeley.
    10. Lori Beaman & Raghabendra Chattopadhyay & Esther Duflo & Rohini Pande & Petia Topalova, 2009. "Powerful Women: Does Exposure Reduce Bias?," The Quarterly Journal of Economics, Oxford University Press, vol. 124(4), pages 1497-1540.
    11. Benjamin A. Olken, 2007. "Monitoring Corruption: Evidence from a Field Experiment in Indonesia," Journal of Political Economy, University of Chicago Press, vol. 115, pages 200-249.
    12. Dean Karlan & John A. List, 2006. "Does Price Matter in Charitable Giving? Evidence from a Large-Scale Natural Field Experiment," Working Papers 1, The Field Experiments Website.
    13. James D. Fearon & Macartan Humphreys & Jeremy M. Weinstein, 2009. "Can Development Aid Contribute to Social Cohesion after Civil War? Evidence from a Field Experiment in Post-conflict Liberia," American Economic Review, American Economic Association, vol. 99(2), pages 287-91, May.
    14. Dan Ariely & Uri Gneezy & George Loewenstein & Nina Mazar, 2005. "Large stakes and big mistakes," Working Papers 05-11, Federal Reserve Bank of Boston.
    15. Vivian Hoffmann, 2009. "Intrahousehold Allocation of Free and Purchased Mosquito Nets," American Economic Review, American Economic Association, vol. 99(2), pages 236-41, May.
    16. Peter Z. Schochet & John Burghardt & Sheena McConnell, 2008. "Does Job Corps Work? Impact Findings from the National Job Corps Study," American Economic Review, American Economic Association, vol. 98(5), pages 1864-86, December.
    17. Paul Schultz, T., 2004. "School subsidies for the poor: evaluating the Mexican Progresa poverty program," Journal of Development Economics, Elsevier, vol. 74(1), pages 199-250, June.
    18. Abhijit V. Banerjee & Shawn Cole & Esther Duflo & Leigh Linden, 2007. "Remedying Education: Evidence from Two Randomized Experiments in India," The Quarterly Journal of Economics, Oxford University Press, vol. 122(3), pages 1235-1264.
    19. Edward Miguel & Michael Kremer & Rebecca Thornton, 2004. "Incentives to learn," Natural Field Experiments 00289, The Field Experiments Website.
    20. Martina Björkman & Jakob Svensson, 2009. "Power to the People: Evidence from a Randomized Field Experiment on Community-Based Monitoring in Uganda," The Quarterly Journal of Economics, Oxford University Press, vol. 124(2), pages 735-769.
    21. Justine S. Hastings & Jeffrey M. Weinstein, 2008. "Information, School Choice, and Academic Achievement: Evidence from Two Experiments," The Quarterly Journal of Economics, Oxford University Press, vol. 123(4), pages 1373-1414.
    22. Bandiera, Oriana & Barankay, Iwan & Rasul, Imran, 2008. "Social Connections and Incentives in the Workplace: Evidence from Personnel Data," IZA Discussion Papers 3917, Institute for the Study of Labor (IZA).
    23. Andreas Lange & Craig Landry & John List & Michael Price & Nicholas Rupp, 2006. "Toward an understanding of the economics of charity: Evidence from a field experiment," Natural Field Experiments 00292, The Field Experiments Website.
    24. Glenn Harrison & John List, 2004. "Field experiments," Artefactual Field Experiments 00058, The Field Experiments Website.
    25. Johanne Boisjoly & Greg J. Duncan & Michael Kremer & Dan M. Levy & Jacque Eccles, 2006. "Empathy or Antipathy? The Impact of Diversity," American Economic Review, American Economic Association, vol. 96(5), pages 1890-1905, December.
    26. Jeffrey R. Kling & Jens Ludwig & Lawrence F. Katz, 2004. "Neighborhood Effects on Crime for Female and Male Youth: Evidence from a Randomized Housing Voucher Experiment," NBER Working Papers 10777, National Bureau of Economic Research, Inc.
    27. repec:oup:restud:v:76:y:2009:i:3:p:1071-1102 is not listed on IDEAS
    28. Peter Z. Schochet & John Burghardt & Sheena McConnell, 2008. "Does Job Corps Work? Impact Findings from the National Job Corps Study," Mathematica Policy Research Reports 5067e4907eba4a4881f510080, Mathematica Policy Research.
    29. Hahn, Jinyong & Hirano, Keisuke & Karlan, Dean, 2009. "Adaptive Experimental Design Using the Propensity Score," Working Papers 59, Yale University, Department of Economics.
    30. Kenneth I. Wolpin & Petra E. Todd, 2006. "Assessing the Impact of a School Subsidy Program in Mexico: Using a Social Experiment to Validate a Dynamic Behavioral Model of Child Schooling and Fertility," American Economic Review, American Economic Association, vol. 96(5), pages 1384-1417, December.
    31. Pascaline Dupas, 2009. "What Matters (and What Does Not) in Households' Decision to Invest in Malaria Prevention?," American Economic Review, American Economic Association, vol. 99(2), pages 224-30, May.
    32. Jeffrey R Kling & Jeffrey B Liebman & Lawrence F Katz, 2007. "Experimental Analysis of Neighborhood Effects," Econometrica, Econometric Society, vol. 75(1), pages 83-119, 01.
    33. Joshua D. Angrist, 2003. "Treatment Effect Heterogeneity in Theory and Practice," NBER Working Papers 9708, National Bureau of Economic Research, Inc.
    34. Skoufias, Emmanuel, 2005. "PROGRESA and its impacts on the welfare of rural households in Mexico:," Research reports 139, International Food Policy Research Institute (IFPRI).
    35. Raymond Fisman & Sheena S. Iyengar & Emir Kamenica & Itamar Simonson, 2006. "Gender Differences in Mate Selection: Evidence From a Speed Dating Experiment," The Quarterly Journal of Economics, Oxford University Press, vol. 121(2), pages 673-697.
    36. Nava Ashraf & Dean Karlan & Wesley Yin, 2006. "Tying Odysseus to the Mast: Evidence From a Commitment Savings Product in the Philippines," The Quarterly Journal of Economics, Oxford University Press, vol. 121(2), pages 635-672.
    37. Esther Duflo & William Gale & Jeffrey Liebman & Peter Orszag & Emmanuel Saez, 2006. "Saving Incentives for Low- and Middle-Income Families: Evidence from a Field Experiment with H&R Block," The Quarterly Journal of Economics, Oxford University Press, vol. 121(4), pages 1311-1346.
    38. Lori A. Beaman & Raghabendra Chattopadhyay & Esther Duflo & Rohini Pande & Petia Topalova, 2008. "Powerful Women: Does Exposure Reduce Bias?," NBER Working Papers 14198, National Bureau of Economic Research, Inc.
    39. Jalan, Jyotsna & Somanathan, E., 2008. "The importance of being informed: Experimental evidence on demand for environmental quality," Journal of Development Economics, Elsevier, vol. 87(1), pages 14-28, August.
    40. van der Laan Mark J. & Dudoit Sandrine & Pollard Katherine S., 2004. "Augmentation Procedures for Control of the Generalized Family-Wise Error Rate and Tail Probabilities for the Proportion of False Positives," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 3(1), pages 1-27, June.
    41. Bandiera, Oriana & Barankay, Iwan & Rasul, Imran, 2006. "Incentives for Managers and Inequality Among Workers: Evidence from a Firm Level Experiment," CEPR Discussion Papers 5649, C.E.P.R. Discussion Papers.
    42. Charness, Gary B & Gneezy, Uri, 2008. "Incentives to Exercise," University of California at Santa Barbara, Economics Working Paper Series qt3tc3j5x7, Department of Economics, UC Santa Barbara.
    43. Ernst Fehr & Lorenz Goette, 2007. "Do Workers Work More if Wages Are High? Evidence from a Randomized Field Experiment," American Economic Review, American Economic Association, vol. 97(1), pages 298-317, March.
    44. Roger Newson & The ALSPAC Study Team, 2003. "Multiple-test procedures and smile plots," Stata Journal, StataCorp LP, vol. 3(2), pages 109-132, June.
    45. Genovese, Christopher R. & Wasserman, Larry, 2006. "Exceedance Control of the False Discovery Proportion," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 1408-1417, December.
    46. Nava Ashraf, 2009. "Spousal Control and Intra-household Decision Making: An Experimental Study in the Philippines," American Economic Review, American Economic Association, vol. 99(4), pages 1245-77, September.
    47. Karen Blumenschein & GlennC. Blomquist & Magnus Johannesson & Nancy Horn & Patricia Freeman, 2008. "Eliciting Willingness to Pay Without Bias: Evidence from a Field Experiment," Economic Journal, Royal Economic Society, vol. 118(525), pages 114-137, 01.
    48. Jen Shang & Rachel Croson, 2009. "A Field Experiment in Charitable Contribution: The Impact of Social Information on the Voluntary Provision of Public Goods," Economic Journal, Royal Economic Society, vol. 119(540), pages 1422-1439, October.
    49. Michael Kremer & Edward Miguel, 2007. "The Illusion of Sustainability," The Quarterly Journal of Economics, Oxford University Press, vol. 122(3), pages 1007-1065.
    50. Jeffrey Carpenter & Jessica Holmes & PeterHans Matthews, 2008. "Charity auctions: a field experiment," Economic Journal, Royal Economic Society, vol. 118(525), pages 92-113, 01.
    51. Gine, Xavier & Yang, Dean, 2007. "Insurance, credit, and technology adoption : field experimental evidence from Malawi," Policy Research Working Paper Series 4425, The World Bank.
    52. Esther Duflo & Michael Kremer & Jonathan Robinson, 2008. "How High Are Rates of Return to Fertilizer? Evidence from Field Experiments in Kenya," American Economic Review, American Economic Association, vol. 98(2), pages 482-88, May.
    Full references (including those not matched with items on IDEAS)

    This item is not listed on Wikipedia, on a reading list or among the top items on IDEAS.

    When requesting a correction, please mention this item's handle: RePEc:han:dpaper:dp-477. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Heidrich, Christian)

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If references are entirely missing, you can add them using this form.

    If the full references list an item that is present in RePEc, but the system did not link to it, you can help with this form.

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    This information is provided to you by IDEAS at the Research Division of the Federal Reserve Bank of St. Louis using RePEc data.