IDEAS home Printed from https://ideas.repec.org/p/zur/econwp/060.html
   My bibliography  Save this paper

Controlling the danger of false discoveries in estimating multiple treatment effects

Author

Listed:
  • Dan Wunderli

Abstract

I expose the risk of false discoveries in the context of multiple treatment effects. A false discovery is a nonexistent effect that is falsely labeled as statistically significant by its individual t-value. Labeling nonexistent effects as statistically significant has wide-ranging academic and policy-related implications, like costly false conclusions from policy evaluations. I eexamine an empirical labor market model by using state-of-the art multiple testing methods and I provide simulation evidence. By merely using individual t-values at conventional significance levels, the risk of labeling probably nonexistent treatment effects as statistically significant is unacceptably high. Individual t-values even label a number of treatment effects as significant, whereas multiple testing indicates false discoveries in these cases. Tests of a joint null hypothesis such as the well-known F-test control the risk of false discoveries only to a limited extent and do not optimally allow for rejecting individual hypotheses. Multiple testing methods control the risk of false discoveries in general while allowing for individual decisions in the sense of rejecting individual hypotheses.

Suggested Citation

  • Dan Wunderli, 2012. "Controlling the danger of false discoveries in estimating multiple treatment effects," ECON - Working Papers 060, Department of Economics - University of Zurich.
  • Handle: RePEc:zur:econwp:060
    as

    Download full text from publisher

    File URL: https://www.zora.uzh.ch/id/eprint/57865/1/econwp060.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Joseph P. Romano & Michael Wolf, 2005. "Stepwise Multiple Testing as Formalized Data Snooping," Econometrica, Econometric Society, vol. 73(4), pages 1237-1282, July.
    2. Jan Boone & Jan C. van Ours, 2006. "Modeling Financial Incentives to Get the Unemployed Back to Work," Journal of Institutional and Theoretical Economics (JITE), Mohr Siebeck, Tübingen, vol. 162(2), pages 227-252, June.
    3. James Heckman & Seong Hyeok Moon & Rodrigo Pinto & Peter Savelyev & Adam Yavitz, 2010. "Analyzing social experiments as implemented: A reexamination of the evidence from the HighScope Perry Preschool Program," Quantitative Economics, Econometric Society, vol. 1(1), pages 1-46, July.
    4. Rafael Lalive & Jan C. van Ours & Josef Zweimüller, 2005. "The Effect Of Benefit Sanctions On The Duration Of Unemployment," Journal of the European Economic Association, MIT Press, vol. 3(6), pages 1386-1417, December.
    5. Heckman, James & Singer, Burton, 1984. "A Method for Minimizing the Impact of Distributional Assumptions in Econometric Models for Duration Data," Econometrica, Econometric Society, vol. 52(2), pages 271-320, March.
    6. James Heckman & Seong Hyeok Moon & Rodrigo Pinto & Peter Savelyev & Adam Yavitz, 2010. "Analyzing social experiments as implemented: evidence from the HighScope Perry Preschool Program," CeMMAP working papers CWP22/10, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Orazio Attanasio & Sarah Cattan & Emla Fitzsimons & Costas Meghir & Marta Rubio-Codina, 2020. "Estimating the Production Function for Human Capital: Results from a Randomized Controlled Trial in Colombia," American Economic Review, American Economic Association, vol. 110(1), pages 48-85, January.
    2. Berger, Eva M. & Fehr, Ernst & Hermes, Henning & Schunk, Daniel & Winkel, Kirsten, 2020. "The Impact of Working Memory Training on Children's Cognitive and Noncognitive Skills," IZA Discussion Papers 13338, Institute of Labor Economics (IZA).
    3. Orazio Attanasio & Sarah Cattan & Emla Fitzsimons & Costas Meghir & Marta Rubio-Codina, 2015. "Estimating the Production Function for Human Capital: Results from a Randomized Control Trial in Colombia," Cowles Foundation Discussion Papers 1987, Cowles Foundation for Research in Economics, Yale University.
    4. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    5. John A. List & Azeem M. Shaikh & Yang Xu, 2019. "Multiple hypothesis testing in experimental economics," Experimental Economics, Springer;Economic Science Association, vol. 22(4), pages 773-793, December.
    6. John A. List & Azeem M. Shaikh & Atom Vayalinkal, 2023. "Multiple testing with covariate adjustment in experimental economics," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(6), pages 920-939, September.
    7. Müller, Kai-Uwe, 2007. "Observed and unobserved determinants of unemployment insurance benefit sanctions in Germany: Evidence from matched individual and regional administrative data," Discussion Papers, Research Unit: Labor Market Policy and Employment SP I 2007-107, WZB Berlin Social Science Center.
    8. Hideo Akabayashi & TIm Ruberg & Chizuru Shikishima & Jun Yamashita, 2023. "Education-Oriented and Care-Oriented Preschools:Implications on Child Development," Keio-IES Discussion Paper Series 2023-009, Institute for Economics Studies, Keio University.
    9. Rodríguez-Planas, Núria & Sanz-de-Galdeano, Anna & Terskaya, Anastasia, 2018. "Independent Thinking and Hard Working, or Caring and Well Behaved? Short- and Long-Term Impacts of Gender Identity Norms," IZA Discussion Papers 11694, Institute of Labor Economics (IZA).
    10. Peter A. Savelyev, 2014. "Psychological Skills, Education, and Longevity of High-Ability Individuals," Vanderbilt University Department of Economics Working Papers 14-00007, Vanderbilt University Department of Economics.
    11. Romano, Joseph P. & Wolf, Michael, 2016. "Efficient computation of adjusted p-values for resampling-based stepdown multiple testing," Statistics & Probability Letters, Elsevier, vol. 113(C), pages 38-40.
    12. Akabayashi, Hideo & Ruberg, Tim & Shikishima, Chizuru & Yamashita, Jun, 2023. "Education-oriented and care-oriented preschools: Implications on child development," Labour Economics, Elsevier, vol. 84(C).
    13. Pedro Carneiro & Oswald Koussihouèdé & Nathalie Lahire & Costas Meghir & Corina Mommaerts, 2020. "School Grants and Education Quality: Experimental Evidence from Senegal," Economica, London School of Economics and Political Science, vol. 87(345), pages 28-51, January.
    14. Sandner, Malte & Cornelissen, Thomas & Jungmann, Tanja & Herrmann, Peggy, 2018. "Evaluating the effects of a targeted home visiting program on maternal and child health outcomes," Journal of Health Economics, Elsevier, vol. 58(C), pages 269-283.
    15. Boone, Jan & Sadrieh, Abdolkarim & van Ours, Jan C., 2009. "Experiments on unemployment benefit sanctions and job search behavior," European Economic Review, Elsevier, vol. 53(8), pages 937-951, November.
    16. Yann Algan & Elizabeth Beasley & Frank Vitaro & Richard Tremblay, 2014. "The Impact of Non-Cognitive Skills Training on Academic and Non-academic Trajectories: From Childhood to Early Adulthood," Working Papers hal-03429906, HAL.
    17. Fabian Kosse & Thomas Deckers & Pia Pinger & Hannah Schildberg-Hörisch & Armin Falk, 2020. "The Formation of Prosociality: Causal Evidence on the Role of Social Environment," Journal of Political Economy, University of Chicago Press, vol. 128(2), pages 434-467.
    18. Patrick Arni & Rafael Lalive & Jan C. Van Ours, 2013. "How Effective Are Unemployment Benefit Sanctions? Looking Beyond Unemployment Exit," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 28(7), pages 1153-1178, November.
    19. Ando, Michihito & Mori, Hiroaki & Yamaguchi, Shintaro, 2022. "Universal early childhood education and adolescent risky behavior," SocArXiv rnkgs, Center for Open Science.
    20. Jorge Luis García & James J. Heckman, 2021. "Early childhood education and life‐cycle health," Health Economics, John Wiley & Sons, Ltd., vol. 30(S1), pages 119-141, November.

    More about this item

    Keywords

    False discoveries; multiple error rates; multiple treatment effects; labor market;
    All these keywords.

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C14 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Semiparametric and Nonparametric Methods: General
    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • C31 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models; Quantile Regressions; Social Interaction Models
    • C41 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods: Special Topics - - - Duration Analysis; Optimal Timing Strategies
    • J08 - Labor and Demographic Economics - - General - - - Labor Economics Policies
    • J64 - Labor and Demographic Economics - - Mobility, Unemployment, Vacancies, and Immigrant Workers - - - Unemployment: Models, Duration, Incidence, and Job Search

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zur:econwp:060. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Severin Oswald (email available below). General contact details of provider: https://edirc.repec.org/data/seizhch.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.