New multiple testing method under no dependency assumption, with application to multiple comparisons problem

Traditional multiple hypotheses testing mainly focuses on constructing stepwise procedures under some error rate control, such as familywise error rate (FWER), false discovery rate, and so forth. However, most of these procedures are obtained in independent case, and when there exists correlation across tests, the dependency may increase or decrease the chance of false rejections. In this paper, a totally different testing method is proposed, which doesn’t focus on specific error control, but pays attention to the overall performance of the collection of hypotheses and the structure utilization among hypotheses. Since the main purpose of multiple testing is to pick out the false ones from the whole hypotheses and present a rejection set, motivated by the principle of simple hypothesis testing, we give the final testing result based on the estimation of the set of all the true null hypotheses. Our method can be applied in any dependent case provided that a reasonable $$p$$ p -value can be obtained for each intersection hypothesis. We illustrate the new procedures with application to multiple comparisons problems. Theoretical results show the consistency of our method, and investigate their FWER behavior. Simulation results suggest that our procedures have a better overall performance than some existing procedures in dependent cases, especially in the total number of type I and type II errors. Copyright Springer-Verlag Berlin Heidelberg 2016