IDEAS home Printed from https://ideas.repec.org/p/fip/feddwp/93384.html
   My bibliography  Save this paper

Empirical Bayes Control of the False Discovery Exceedance

Author

Abstract

In sparse large-scale testing problems where the false discovery proportion (FDP) is highly variable, the false discovery exceedance (FDX) provides a valuable alternative to the widely used false discovery rate (FDR). We develop an empirical Bayes approach to controlling the FDX. We show that for independent hypotheses from a two-group model and dependent hypotheses from a Gaussian model fulfilling the exchangeability condition, an oracle decision rule based on ranking and thresholding the local false discovery rate (lfdr) is optimal in the sense that the power is maximized subject to FDX constraint. We propose a data-driven FDX procedure that emulates the oracle via carefully designed computational shortcuts. We investigate the empirical performance of the proposed method using simulations and illustrate the merits of FDX control through an application for identifying abnormal stock trading strategies.

Suggested Citation

  • Pallavi Basu & Luella Fu & Alessio Saretto & Wenguang Sun, 2021. "Empirical Bayes Control of the False Discovery Exceedance," Working Papers 2115, Federal Reserve Bank of Dallas.
    • Handle: RePEc:fip:feddwp:93384
    DOI: 10.24149/wp2115
    as

    Download full text from publisher

    File URL: https://www.dallasfed.org/-/media/documents/research/papers/2021/wp2115.pdf
    File Function: Full text
    Download Restriction: no
    File URL: https://libkey.io/10.24149/wp2115?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Sun, Wenguang & Cai, T. Tony, 2007. "Oracle and Adaptive Compound Decision Rules for False Discovery Rate Control," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 901-912, September.
    2. Genovese, Christopher R. & Wasserman, Larry, 2006. "Exceedance Control of the False Discovery Proportion," Journal of the American Statistical Association, American Statistical Association, vol. 101, pages 1408-1417, December.
    3. van der Laan Mark J. & Birkner Merrill D. & Hubbard Alan E., 2005. "Empirical Bayes and Resampling Based Multiple Testing Procedure Controlling Tail Probability of the Proportion of False Positives," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 4(1), pages 1-32, October.
    4. Romano, Joseph P. & Shaikh, Azeem M. & Wolf, Michael, 2008. "Formalized Data Snooping Based On Generalized Error Rates," Econometric Theory, Cambridge University Press, vol. 24(2), pages 404-447, April.
    5. van der Laan Mark J. & Dudoit Sandrine & Pollard Katherine S., 2004. "Augmentation Procedures for Control of the Generalized Family-Wise Error Rate and Tail Probabilities for the Proportion of False Positives," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 3(1), pages 1-27, June.
    6. Andrew Y. Chen & Tom Zimmermann, 2022. "Open Source Cross-Sectional Asset Pricing," Critical Finance Review, now publishers, vol. 11(2), pages 207-264, May.
    7. Fama, Eugene F. & French, Kenneth R., 2015. "A five-factor asset pricing model," Journal of Financial Economics, Elsevier, vol. 116(1), pages 1-22.
    8. Mark van der Laan & Sandrine Dudoit & Katherine Pollard, 2004. "Multiple Testing. Part III. Procedures for Control of the Generalized Family-Wise Error Rate and Proportion of False Positives," U.C. Berkeley Division of Biostatistics Working Paper Series 1140, Berkeley Electronic Press.
    9. Efron, Bradley, 2009. "Empirical Bayes Estimates for Large-Scale Prediction Problems," Journal of the American Statistical Association, American Statistical Association, vol. 104(487), pages 1015-1028.
    10. Gordon, Alexander Y. & Salzman, Peter, 2008. "Optimality of the Holm procedure among general step-down multiple testing procedures," Statistics & Probability Letters, Elsevier, vol. 78(13), pages 1878-1884, September.
    11. Efron B. & Tibshirani R. & Storey J.D. & Tusher V., 2001. "Empirical Bayes Analysis of a Microarray Experiment," Journal of the American Statistical Association, American Statistical Association, vol. 96, pages 1151-1160, December.
    12. Wenguang Sun & T. Tony Cai, 2009. "Large‐scale multiple testing under dependence," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 71(2), pages 393-424, April.
    13. Alessio Farcomeni, 2009. "Generalized Augmentation to Control the False Discovery Exceedance in Multiple Testing," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 36(3), pages 501-517, September.
    14. Pallavi Basu & T. Tony Cai & Kiranmoy Das & Wenguang Sun, 2018. "Weighted False Discovery Rate Control in Large-Scale Multiple Testing," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(523), pages 1172-1183, July.
    15. Tarun Chordia & Amit Goyal & Alessio Saretto & Andrew KarolyiEditor, 2020. "Anomalies and False Rejections," Review of Finance, European Finance Association, vol. 33(5), pages 2134-2179.
    16. Joseph P. Romano & Michael Wolf, "undated". "Control of Generalized Error Rates in Multiple Testing," IEW - Working Papers 245, Institute for Empirical Research in Economics - University of Zurich.
    17. Carhart, Mark M, 1997. "On Persistence in Mutual Fund Performance," Journal of Finance, American Finance Association, vol. 52(1), pages 57-82, March.
    18. Ruth Heller & Saharon Rosset, 2021. "Optimal control of false discovery criteria in the two‐group model," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 83(1), pages 133-155, February.
    19. Fraley C. & Raftery A.E., 2002. "Model-Based Clustering, Discriminant Analysis, and Density Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 97, pages 611-631, June.
    20. Guo Wenge & Romano Joseph, 2007. "A Generalized Sidak-Holm Procedure and Control of Generalized Error Rates under Independence," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 6(1), pages 1-35, January.
    21. Efron, Bradley, 2004. "Large-Scale Simultaneous Hypothesis Testing: The Choice of a Null Hypothesis," Journal of the American Statistical Association, American Statistical Association, vol. 99, pages 96-104, January.
    22. M. Perone Pacifico & C. Genovese & I. Verdinelli & L. Wasserman, 2004. "False Discovery Control for Random Fields," Journal of the American Statistical Association, American Statistical Association, vol. 99, pages 1002-1014, December.
    23. Tarun Chordia & Amit Goyal & Alessio Saretto, 2020. "Anomalies and False Rejections," The Review of Financial Studies, Society for Financial Studies, vol. 33(5), pages 2134-2179.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Alessio Farcomeni, 2009. "Generalized Augmentation to Control the False Discovery Exceedance in Multiple Testing," Scandinavian Journal of Statistics, Danish Society for Theoretical Statistics;Finnish Statistical Society;Norwegian Statistical Association;Swedish Statistical Association, vol. 36(3), pages 501-517, September.
    2. Wang, Li & Xu, Xingzhong, 2012. "Step-up procedure controlling generalized family-wise error rate," Statistics & Probability Letters, Elsevier, vol. 82(4), pages 775-782.
    3. Joseph P. Romano & Azeem M. Shaikh & Michael Wolf, 2010. "Hypothesis Testing in Econometrics," Annual Review of Economics, Annual Reviews, vol. 2(1), pages 75-104, September.
    4. Andrew Y. Chen, 2021. "The Limits of p‐Hacking: Some Thought Experiments," Journal of Finance, American Finance Association, vol. 76(5), pages 2447-2480, October.
    5. Mathur, Maya B & VanderWeele, Tyler, 2018. "Statistical methods for evidence synthesis," Thesis Commons kd6ja, Center for Open Science.
    6. Li Wang, 2019. "Weighted multiple testing procedure for grouped hypotheses with k-FWER control," Computational Statistics, Springer, vol. 34(2), pages 885-909, June.
    7. Habiger, Joshua D. & Peña, Edsel A., 2014. "Compound p-value statistics for multiple testing procedures," Journal of Multivariate Analysis, Elsevier, vol. 126(C), pages 153-166.
    8. Joseph P. Romano & Michael Wolf, 2008. "Balanced Control of Generalized Error Rates," IEW - Working Papers 379, Institute for Empirical Research in Economics - University of Zurich.
    9. Joseph Romano & Azeem Shaikh & Michael Wolf, 2008. "Control of the false discovery rate under dependence using the bootstrap and subsampling," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 17(3), pages 417-442, November.
    10. Andrew Y. Chen & Tom Zimmermann, 2022. "Publication Bias in Asset Pricing Research," Papers 2209.13623, arXiv.org, revised Sep 2023.
    11. James W. Kolari & Jianhua Z. Huang & Wei Liu & Huiling Liao, 2022. "Further Tests of the ZCAPM Asset Pricing Model," JRFM, MDPI, vol. 15(3), pages 1-23, March.
    12. Montazeri Zahra & Yanofsky Corey M. & Bickel David R., 2010. "Shrinkage Estimation of Effect Sizes as an Alternative to Hypothesis Testing Followed by Estimation in High-Dimensional Biology: Applications to Differential Gene Expression," Statistical Applications in Genetics and Molecular Biology, De Gruyter, vol. 9(1), pages 1-33, June.
    13. Qiao, Zhuo & Wang, Yan & Lam, Keith S.K., 2022. "New evidence on Bayesian tests of global factor pricing models," Journal of Empirical Finance, Elsevier, vol. 68(C), pages 160-172.
    14. Andrew Y. Chen, 2022. "Most claimed statistical findings in cross-sectional return predictability are likely true," Papers 2206.15365, arXiv.org, revised Mar 2024.
    15. Söhnke M. Bartram & Harald Lohre & Peter F. Pope & Ananthalakshmi Ranganathan, 2021. "Navigating the factor zoo around the world: an institutional investor perspective," Journal of Business Economics, Springer, vol. 91(5), pages 655-703, July.
    16. Cerioli, Andrea & Farcomeni, Alessio, 2011. "Error rates for multivariate outlier detection," Computational Statistics & Data Analysis, Elsevier, vol. 55(1), pages 544-553, January.
    17. T. Tony Cai & Wenguang Sun & Weinan Wang, 2019. "Covariate‐assisted ranking and screening for large‐scale two‐sample inference," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 81(2), pages 187-234, April.
    18. Kolari, James W. & Huang, Jianhua Z. & Butt, Hilal Anwar & Liao, Huiling, 2022. "International tests of the ZCAPM asset pricing model," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 79(C).
    19. Yu-Chin Hsu & Hsiou-Wei Lin & Kendro Vincent, 2017. "Do Cross-Sectional Stock Return Predictors Pass the Test without Data-Snooping Bias?," IEAS Working Paper : academic research 17-A003, Institute of Economics, Academia Sinica, Taipei, Taiwan.
    20. Cakici, Nusret & Zaremba, Adam, 2023. "Recency bias and the cross-section of international stock returns," Journal of International Financial Markets, Institutions and Money, Elsevier, vol. 84(C).

    More about this item

    Keywords

    Cautious Data Mining; False Discovery Exceedance Control; Local False Discovery Rates; Multiple Hypotheses Testing; Poisson Binomial Distribution; Trading Strategies;
    All these keywords.

    JEL classification:

    • C11 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Bayesian Analysis: General
    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C15 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Statistical Simulation Methods: General

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:fip:feddwp:93384. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Amy Chapman (email available below). General contact details of provider: https://edirc.repec.org/data/frbdaus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.