IDEAS home Printed from https://ideas.repec.org/p/zur/econwp/060.html
   My bibliography  Save this paper

Controlling the danger of false discoveries in estimating multiple treatment effects

Author

Listed:
  • Dan Wunderli

Abstract

I expose the risk of false discoveries in the context of multiple treatment effects. A false discovery is a nonexistent effect that is falsely labeled as statistically significant by its individual t-value. Labeling nonexistent effects as statistically significant has wide-ranging academic and policy-related implications, like costly false conclusions from policy evaluations. I eexamine an empirical labor market model by using state-of-the art multiple testing methods and I provide simulation evidence. By merely using individual t-values at conventional significance levels, the risk of labeling probably nonexistent treatment effects as statistically significant is unacceptably high. Individual t-values even label a number of treatment effects as significant, whereas multiple testing indicates false discoveries in these cases. Tests of a joint null hypothesis such as the well-known F-test control the risk of false discoveries only to a limited extent and do not optimally allow for rejecting individual hypotheses. Multiple testing methods control the risk of false discoveries in general while allowing for individual decisions in the sense of rejecting individual hypotheses.

Suggested Citation

  • Dan Wunderli, 2012. "Controlling the danger of false discoveries in estimating multiple treatment effects," ECON - Working Papers 060, Department of Economics - University of Zurich.
    • Handle: RePEc:zur:econwp:060
    as

    Download full text from publisher

    File URL: http://www.econ.uzh.ch/static/wp/econwp060.pdf
    Download Restriction: no

    References listed on IDEAS

    as
    1. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    2. Heckman, James & Singer, Burton, 1984. "A Method for Minimizing the Impact of Distributional Assumptions in Econometric Models for Duration Data," Econometrica, Econometric Society, vol. 52(2), pages 271-320, March.
    3. Jan Boone & Jan C. van Ours, 2000. "Modeling Financial Incentives to Get Unemployed Back to Work," Econometric Society World Congress 2000 Contributed Papers 0973, Econometric Society.
    4. Rafael Lalive & Jan C. van Ours & Josef Zweimüller, 2005. "The Effect Of Benefit Sanctions On The Duration Of Unemployment," Journal of the European Economic Association, MIT Press, vol. 3(6), pages 1386-1417, December.
    5. James Heckman & Seong Hyeok Moon & Rodrigo Pinto & Peter Savelyev & Adam Yavitz, 2010. "Analyzing social experiments as implemented: A reexamination of the evidence from the HighScope Perry Preschool Program," Quantitative Economics, Econometric Society, vol. 1(1), pages 1-46, July.
    Full references (including those not matched with items on IDEAS)

    More about this item

    Keywords

    False discoveries; multiple error rates; multiple treatment effects; labor market;

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C14 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Semiparametric and Nonparametric Methods: General
    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • C31 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models; Quantile Regressions; Social Interaction Models
    • C41 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - Duration Analysis; Optimal Timing Strategies
    • J08 - Labor and Demographic Economics - - General - - - Labor Economics Policies
    • J64 - Labor and Demographic Economics - - Mobility, Unemployment, Vacancies, and Immigrant Workers - - - Unemployment: Models, Duration, Incidence, and Job Search

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zur:econwp:060. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Marita Kieser). General contact details of provider: http://edirc.repec.org/data/seizhch.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.