Controlling the number of significant effects in multiple testing

In multiple testing, several criteria to control for type I errors exist. The false discovery rate, which evaluates the expected proportion of false discoveries among the rejected null hypotheses, has become the standard approach in this setting. However, false discovery rate control may be too conservative when the effects are weak, that is, when the true alternative hypotheses are close to their corresponding nulls. In this article, we alternatively propose to focus on the number of significant effects, where ’significant’ refers to a pre-specified threshold γ. In particular, a (1−α)-lower confidence bound N for the number of non true null hypotheses with p-value below γ is provided. When one rejects the nulls corresponding to the N smallest p-values, the probability that the number of false positives exceeds the number of false negatives among the significant effects is bounded by α. The method aims to improve the statistical power in the multiple testing setup while avoiding an unjustifiably large amount of rejected nulls. Relative merits of the proposed criterion are discussed. Procedures to control for the number of significant effects in practice are introduced and investigated both theoretically and through simulations. Illustrative real data applications are given.