IDEAS home Printed from https://ideas.repec.org/a/taf/jnlasa/v114y2019i527p1291-1304.html
   My bibliography  Save this article

Multiple Testing When Many p-Values are Uniformly Conservative, with Application to Testing Qualitative Interaction in Educational Interventions

Author

Listed:
  • Qingyuan Zhao
  • Dylan S. Small
  • Weijie Su

Abstract

In the evaluation of treatment effects, it is of major policy interest to know if the treatment is beneficial for some and harmful for others, a phenomenon known as qualitative interaction. We formulate this question as a multiple testing problem with many conservative null p-values, in which the classical multiple testing methods may lose power substantially. We propose a simple technique—conditioning—to improve the power. A crucial assumption we need is uniform conservativeness, meaning for any conservative p-value p, the conditional distribution (p/τ) | p ⩽ τ is stochastically larger than the uniform distribution on (0, 1) for any τ. We show this property holds for one-sided tests in a one-dimensional exponential family (e.g., testing for qualitative interaction) as well as testing |μ| ⩽ η using a statistic Y ∼ N(μ, 1) (e.g., testing for practical importance with threshold η). We propose an adaptive method to select the threshold τ. Our theoretical and simulation results suggest that the proposed tests gain significant power when many p-values are uniformly conservative and lose little power when no p-value is uniformly conservative. We apply our method to two educational intervention datasets. Supplementary materials for this article are available online.

Suggested Citation

  • Qingyuan Zhao & Dylan S. Small & Weijie Su, 2019. "Multiple Testing When Many p-Values are Uniformly Conservative, with Application to Testing Qualitative Interaction in Educational Interventions," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 114(527), pages 1291-1304, July.
  • Handle: RePEc:taf:jnlasa:v:114:y:2019:i:527:p:1291-1304
    DOI: 10.1080/01621459.2018.1497499
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1080/01621459.2018.1497499
    Download Restriction: Access to full text is restricted to subscribers.

    File URL: https://libkey.io/10.1080/01621459.2018.1497499?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Anh-Tuan Hoang & Thorsten Dickhaus, 2022. "On the usage of randomized p-values in the Schweder–Spjøtvoll estimator," Annals of the Institute of Statistical Mathematics, Springer;The Institute of Statistical Mathematics, vol. 74(2), pages 289-319, April.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:taf:jnlasa:v:114:y:2019:i:527:p:1291-1304. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Chris Longhurst (email available below). General contact details of provider: http://www.tandfonline.com/UASA20 .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.