IDEAS home Printed from https://ideas.repec.org/a/bpj/sagmbi/v5y2006i1n19.html
   My bibliography  Save this article

A Method to Increase the Power of Multiple Testing Procedures Through Sample Splitting

Author

Listed:
  • Rubin Daniel

    (University of California, Berkeley)

  • Dudoit Sandrine

    (University of California, Berkeley)

  • van der Laan Mark

    (University of California, Berkeley)

Abstract

Consider the standard multiple testing problem where many hypotheses are to be tested, each hypothesis is associated with a test statistic, and large test statistics provide evidence against the null hypotheses. One proposal to provide probabilistic control of Type-I errors is the use of procedures ensuring that the expected number of false positives does not exceed a user-supplied threshold. Among such multiple testing procedures, we derive the most powerful method, meaning the test statistic cutoffs that maximize the expected number of true positives. Unfortunately, these optimal cutoffs depend on the true unknown data generating distribution, so could never be used in a practical setting. We instead consider splitting the sample so that the optimal cutoffs are estimated from a portion of the data, and then testing on the remaining data using these estimated cutoffs. When the null distributions for all test statistics are the same, the obvious way to control the expected number of false positives would be to use a common cutoff for all tests. In this work, we consider the common cutoff method as a benchmark multiple testing procedure. We show that in certain circumstances the use of estimated optimal cutoffs via sample splitting can dramatically outperform this benchmark method, resulting in increased true discoveries, while retaining Type-I error control. This paper is an updated version of the work presented in Rubin et al. (2005), later expanded upon by Wasserman and Roeder (2006).

Suggested Citation

  • Rubin Daniel & Dudoit Sandrine & van der Laan Mark, 2006. "A Method to Increase the Power of Multiple Testing Procedures Through Sample Splitting," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 5(1), pages 1-20, August.
  • Handle: RePEc:bpj:sagmbi:v:5:y:2006:i:1:n:19
    DOI: 10.2202/1544-6115.1148
    as

    Download full text from publisher

    File URL: https://doi.org/10.2202/1544-6115.1148
    Download Restriction: For access to full text, subscription to the journal or payment for the individual article is required.

    File URL: https://libkey.io/10.2202/1544-6115.1148?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Dudoit Sandrine & van der Laan Mark J. & Pollard Katherine S., 2004. "Multiple Testing. Part I. Single-Step Procedures for Control of General Type I Error Rates," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 3(1), pages 1-71, June.
    2. Sandrine Dudoit & Mark van der Laan & Katherine Pollard, 2004. "Multiple Testing. Part I. Single-Step Procedures for Control of General Type I Error Rates," U.C. Berkeley Division of Biostatistics Working Paper Series 1137, Berkeley Electronic Press.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Edgar Dobriban & Kristen Fortney & Stuart K. Kim & Art B. Owen, 2015. "Optimal multiple testing under a Gaussian prior on the effect sizes," Biometrika, Biometrika Trust, vol. 102(4), pages 753-766.
    2. Nikolaos Ignatiadis & Wolfgang Huber, 2021. "Covariate powered cross‐weighted multiple testing," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 83(4), pages 720-751, September.
    3. Habiger, Joshua D. & Peña, Edsel A., 2014. "Compound p-value statistics for multiple testing procedures," Journal of Multivariate Analysis, Elsevier, vol. 126(C), pages 153-166.
    4. Miecznikowski, Jeffrey C. & Gold, David & Shepherd, Lori & Liu, Song, 2011. "Deriving and comparing the distribution for the number of false positives in single step methods to control k-FWER," Statistics & Probability Letters, Elsevier, vol. 81(11), pages 1695-1705, November.
    5. Shi, Chengchun & Li, Lexin, 2022. "Testing mediation effects using logic of Boolean matrices," LSE Research Online Documents on Economics 108881, London School of Economics and Political Science, LSE Library.
    6. T. Tony Cai & Wenguang Sun & Weinan Wang, 2019. "Covariate‐assisted ranking and screening for large‐scale two‐sample inference," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 81(2), pages 187-234, April.
    7. Simeone Marino & Yi Zhao & Nina Zhou & Yiwang Zhou & Arthur W Toga & Lu Zhao & Yingsi Jian & Yichen Yang & Yehu Chen & Qiucheng Wu & Jessica Wild & Brandon Cummings & Ivo D Dinov, 2020. "Compressive Big Data Analytics: An ensemble meta-algorithm for high-dimensional multisource datasets," PLOS ONE, Public Library of Science, vol. 15(8), pages 1-21, August.
    8. Jordà, Òscar & Knüppel, Malte & Marcellino, Massimiliano, 2013. "Empirical simultaneous prediction regions for path-forecasts," International Journal of Forecasting, Elsevier, vol. 29(3), pages 456-468.
    9. Michael A. Langston & Robert S. Levine & Barbara J. Kilbourne & Gary L. Rogers & Anne D. Kershenbaum & Suzanne H. Baktash & Steven S. Coughlin & Arnold M. Saxton & Vincent K. Agboto & Darryl B. Hood &, 2014. "Scalable Combinatorial Tools for Health Disparities Research," IJERPH, MDPI, vol. 11(10), pages 1-25, October.
    10. Kang Guolian & Ye Keying & Liu Nianjun & Allison David B. & Gao Guimin, 2009. "Weighted Multiple Hypothesis Testing Procedures," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 8(1), pages 1-24, April.
    11. Jelle J Goeman & Aldo Solari, 2024. "On selection and conditioning in multiple testing and selective inference," Biometrika, Biometrika Trust, vol. 111(2), pages 393-416.
    12. DiCiccio, Cyrus J. & DiCiccio, Thomas J. & Romano, Joseph P., 2020. "Exact tests via multiple data splitting," Statistics & Probability Letters, Elsevier, vol. 166(C).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hossain, Ahmed & Beyene, Joseph & Willan, Andrew R. & Hu, Pingzhao, 2009. "A flexible approximate likelihood ratio test for detecting differential expression in microarray data," Computational Statistics & Data Analysis, Elsevier, vol. 53(10), pages 3685-3695, August.
    2. Christina C. Bartenschlager & Michael Krapp, 2015. "Theorie und Methoden multipler statistischer Vergleiche," AStA Wirtschafts- und Sozialstatistisches Archiv, Springer;Deutsche Statistische Gesellschaft - German Statistical Society, vol. 9(2), pages 107-129, November.
    3. Merrill Birkner & Sandra Sinisi & Mark van der Laan, 2004. "Multiple Testing and Data Adaptive Regression: An Application to HIV-1 Sequence Data," U.C. Berkeley Division of Biostatistics Working Paper Series 1161, Berkeley Electronic Press.
    4. Miecznikowski, Jeffrey C. & Gold, David & Shepherd, Lori & Liu, Song, 2011. "Deriving and comparing the distribution for the number of false positives in single step methods to control k-FWER," Statistics & Probability Letters, Elsevier, vol. 81(11), pages 1695-1705, November.
    5. Yoshiyuki Ninomiya & Hironori Fujisawa, 2007. "A Conservative Test for Multiple Comparison Based on Highly Correlated Test Statistics," Biometrics, The International Biometric Society, vol. 63(4), pages 1135-1142, December.
    6. Miecznikowski Jeffrey C. & Gaile Daniel P., 2014. "A novel characterization of the generalized family wise error rate using empirical null distributions," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 13(3), pages 299-322, June.
    7. Schumi Jennifer & DiRienzo A. Gregory & DeGruttola Victor, 2008. "Testing for Associations with Missing High-Dimensional Categorical Covariates," The International Journal of Biostatistics, De Gruyter, vol. 4(1), pages 1-19, September.
    8. Joseph P. Romano & Michael Wolf, 2008. "Balanced Control of Generalized Error Rates," IEW - Working Papers 379, Institute for Empirical Research in Economics - University of Zurich.
    9. van der Laan Mark J. & Hubbard Alan E., 2006. "Quantile-Function Based Null Distribution in Resampling Based Multiple Testing," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 5(1), pages 1-30, May.
    10. Christopher J. Bennett, 2009. "p-Value Adjustments for Asymptotic Control of the Generalized Familywise Error Rate," Vanderbilt University Department of Economics Working Papers 0905, Vanderbilt University Department of Economics.
    11. Yang Yang & Victor DeGruttola, 2008. "Resampling-Based Multiple Testing Methods with Covariate Adjustment: Application to Investigation of Antiretroviral Drug Susceptibility," Biometrics, The International Biometric Society, vol. 64(2), pages 329-336, June.

    More about this item

    Keywords

    multiple testing;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bpj:sagmbi:v:5:y:2006:i:1:n:19. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Peter Golla (email available below). General contact details of provider: https://www.degruyter.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.