IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/29843.html
   My bibliography  Save this paper

Highly Powered Analysis Plans

Author

Listed:
  • Michael L. Anderson
  • Jeremy Magruder

Abstract

Formal analysis plans limit false discoveries by registering and multiplicity adjusting statistical tests. As each registered test reduces power on other tests, researchers prune hypotheses based on prior knowledge, often by combining related indicators into evenly-weighted indices. We propose two improvements to maximize learning within these types of analysis plans. First, we develop data-driven optimized indices that can yield more powerful tests than evenly-weighted indices. Second, we discuss organizing the logical structure of an analysis plan into a gated tree that directs type I error towards these high-powered tests. In simulations we show that researchers may prefer these "optimus gates" across a wide range of data-generating processes. We then assess our strategy using the community-driven development (CDD) application from Casey et al. (2012) and the Oregon Health Insurance Experiment from Finkelstein et al. (2012). We find substantial power gains in both applications, meaningfully changing the conclusions of Casey et al. (2012).

Suggested Citation

  • Michael L. Anderson & Jeremy Magruder, 2022. "Highly Powered Analysis Plans," NBER Working Papers 29843, National Bureau of Economic Research, Inc.
    • Handle: RePEc:nbr:nberwo:29843
    Note: DEV LS PE
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w29843.pdf
    Download Restriction: no
    ---><---

    More about this item

    JEL classification:

    • C12 - Mathematical and Quantitative Methods - - Econometric and Statistical Methods and Methodology: General - - - Hypothesis Testing: General
    • C55 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Large Data Sets: Modeling and Analysis
    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • C9 - Mathematical and Quantitative Methods - - Design of Experiments
    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • O1 - Economic Development, Innovation, Technological Change, and Growth - - Economic Development

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:29843. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no bibliographic references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.