IDEAS home Printed from
   My bibliography  Save this paper

Consonance and the closure method in multiple testing


  • Joseph P. Romano
  • Azeem M. Shaikh
  • Michael Wolf


Consider the problem of testing s hypotheses simultaneously. In order to deal with the multiplicity problem, the classical approach is to restrict attention to procedures that control the familywise error rate (FWE). Typically, it is known how to construct tests of the individual hypotheses, and the problem is how to combine them into a multiple testing procedure that controls the FWE. The closure method of Marcus et al. (1976), in fact, reduces the problem of constructing multiple test procedures which control the FWE to the construction of single tests which control the usual probability of a Type 1 error. The purpose of this paper is to examine the closure method with emphasis on the concepts of coherence and consonance. It was shown by Sonnemann and Finner (1988) that any incoherent procedure can be replaced by a coherent one which is at least as good. The main point of this paper is to show a similar result for dissonant and consonant procedures. We illustrate the idea of how a dissonant procedure can be strictly improved by a consonant procedure in the sense of increasing the probability of detecting a false null hypothesis while maintaining control of the FWE. We then show how consonance can be used in the construction of some optimal maximin procedures.

Suggested Citation

  • Joseph P. Romano & Azeem M. Shaikh & Michael Wolf, 2009. "Consonance and the closure method in multiple testing," IEW - Working Papers 446, Institute for Empirical Research in Economics - University of Zurich.
  • Handle: RePEc:zur:iewwpx:446

    Download full text from publisher

    File URL:
    Download Restriction: no

    Other versions of this item:

    References listed on IDEAS

    1. Richard M. Bittman & Joseph P. Romano & Carlos Vallarino & Michael Wolf, 2009. "Optimal testing of multiple hypotheses with common effect direction," Biometrika, Biometrika Trust, vol. 96(2), pages 399-410.
    Full references (including those not matched with items on IDEAS)


    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.

    Cited by:

    1. David M. Ritzwoller & Joseph P. Romano, 2019. "Uncertainty in the Hot Hand Fallacy: Detecting Streaky Alternatives in Random Bernoulli Sequences," Papers 1908.01406,, revised Aug 2019.
    2. Zeng-Hua Lu, 2016. "Extended MaxT Tests of One-Sided Hypotheses," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 111(513), pages 423-437, March.
    3. Zeng-Hua Lu, 2019. "Extended MinP Tests of Multiple Hypotheses," Papers 1911.04696,
    4. Chung, EunYi & Romano, Joseph P., 2016. "Multivariate and multiple permutation tests," Journal of Econometrics, Elsevier, vol. 193(1), pages 76-91.
    5. Michael Rosenblum & Han Liu & En-Hsu Yen, 2014. "Optimal Tests of Treatment Effects for the Overall Population and Two Subpopulations in Randomized Trials, Using Sparse Linear Programming," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 109(507), pages 1216-1228, September.

    More about this item


    Multiple testing; closure method; coherence; consonance; familywise error rate;

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C14 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Semiparametric and Nonparametric Methods: General

    NEP fields

    This paper has been announced in the following NEP Reports:


    Access and download statistics


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:zur:iewwpx:446. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Marita Kieser). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.