Correlation Between the True and False Discoveries in a Positively Dependent Multiple Comparison Problem

Testing multiple hypotheses when observations are positively correlated is very common in practice. The dependence between observations can induce dependence between test statistics and distort the joint distribution of the true and false positives. It has a profound impact on the performance of common multiple testing procedures. While the marginal statistical properties of the true and false discoveries such as their means and variances have been extensively studied in the past, their correlation remains unknown. By conducting a thorough simulation study, we find that the true and false positives are likely to be negatively correlated if testing power is high and the opposite holds true—they are likely to be positively correlated if testing power is low. The fact that positive dependence between observations can induce negative correlation between the true and false discoveries may assist researchers in designing multiple testing procedures for dependent tests in the future.