IDEAS home Printed from
   My bibliography  Save this article

Multiple Testing And Heterogeneous Treatment Effects: Re‐Evaluating The Effect Of Progresa On School Enrollment


  • Soohyung Lee
  • Azeem M. Shaikh


SUMMARY The effect of a program or treatment may vary according to observed characteristics. In such a setting, it may not only be of interest to determine whether the program or treatment has an effect on some sub‐population defined by these observed characteristics, but also to determine for which sub‐populations, if any, there is an effect. This paper treats this problem as a multiple testing problem in which each null hypothesis in the family of null hypotheses specifies whether the program has an effect on the outcome of interest for a particular sub‐population. We develop our methodology in the context of PROGRESA, a large‐scale poverty‐reduction program in Mexico. In our application, the outcome of interest is the school enrollment rate and the sub‐populations are defined by gender and highest grade completed. Under weak assumptions, the testing procedure we construct controls the familywise error rate—the probability of even one false rejection—in finite samples. Similar to earlier studies, we find that the program has a significant effect on the school enrollment rate, but only for a much smaller number of sub‐populations when compared to results that do not adjust for multiple testing. Copyright © 2013 John Wiley & Sons, Ltd.

Suggested Citation

  • Soohyung Lee & Azeem M. Shaikh, 2014. "Multiple Testing And Heterogeneous Treatment Effects: Re‐Evaluating The Effect Of Progresa On School Enrollment," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(4), pages 612-626, June.
  • Handle: RePEc:wly:japmet:v:29:y:2014:i:4:p:612-626

    Download full text from publisher

    File URL:
    Download Restriction: no


    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.

    Cited by:

    1. Sokbae Lee & Ryo Okui & Yoon†Jae Whang, 2017. "Doubly robust uniform confidence band for the conditional average treatment effect function," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 32(7), pages 1207-1225, November.
    2. Arduini, Tiziano & Patacchini, Eleonora & Rainone, Edoardo, 2019. "Treatment Effects with Heterogeneous Externalities," CEPR Discussion Papers 13781, C.E.P.R. Discussion Papers.
    3. Jonathan M.V. Davis & Sara B. Heller, 2017. "Rethinking the Benefits of Youth Employment Programs: The Heterogeneous Effects of Summer Jobs," NBER Working Papers 23443, National Bureau of Economic Research, Inc.
    4. Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2015. "Inference under covariate-adaptive randomization," CeMMAP working papers CWP45/15, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    5. Weijia (Daisy) Dai & Michael Luca, 2016. "Effectiveness of Paid Search Advertising: Experimental Evidence," Harvard Business School Working Papers 17-025, Harvard Business School.
    6. Aminou Arouna & Jeffrey D. Michler & Jourdain C. Lokossou, 2019. "Contract Farming and Rural Transformation: Evidence from a Field Experiment in Benin," NBER Working Papers 25665, National Bureau of Economic Research, Inc.
    7. Bedoya Arguelles,Guadalupe & Bittarello,Luca & Davis,Jonathan Martin Villars & Mittag,Nikolas Karl & Bedoya Arguelles,Guadalupe & Bittarello,Luca & Davis,Jonathan Martin Villars & Mittag,Nikolas Karl, 2017. "Distributional impact analysis: toolkit and illustrations of impacts beyond the average treatment effect," Policy Research Working Paper Series 8139, The World Bank.
    8. Garret Christensen & Edward Miguel, 2018. "Transparency, Reproducibility, and the Credibility of Economics Research," Journal of Economic Literature, American Economic Association, vol. 56(3), pages 920-980, September.
    9. Azeem M. Shaikh, 2019. "Inference in Experiments with Matched Pairs," CeMMAP working papers CWP19/19, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    10. John A. List & Azeem M. Shaikh & Yang Xu, 2019. "Multiple hypothesis testing in experimental economics," Experimental Economics, Springer;Economic Science Association, vol. 22(4), pages 773-793, December.
    11. Fischer, Stefanie & Royer, Heather & White, Corey, 2018. "The impacts of reduced access to abortion and family planning services on abortions, births, and contraceptive purchases," Journal of Public Economics, Elsevier, vol. 167(C), pages 43-68.
    12. Timothy B. Armstrong & Shu Shen, 2013. "Inference on Optimal Treatment Assignments," Cowles Foundation Discussion Papers 1927, Cowles Foundation for Research in Economics, Yale University.
    13. Lehrer, Steven F. & Pohl, R. Vincent & Song, Kyungchul, 2018. "Multiple Testing and the Distributional Effects of Accountability Incentives in Education," MPRA Paper 89532, University Library of Munich, Germany.
    14. Brennan S. Thompson & Matthew D. Webb, 2015. "A Simple, Graphical Approach to Comparing Multiple Treatments," Working Papers 063, Ryerson University, Department of Economics, revised Mar 2017.
    15. Eszter Czibor & David Jimenez‐Gomez & John A. List, 2019. "The Dozen Things Experimental Economists Should Do (More of)," Southern Economic Journal, John Wiley & Sons, vol. 86(2), pages 371-432, October.
    16. Brennan S Thompson & Matthew D Webb, 2019. "A simple, graphical approach to comparing multiple treatments," Econometrics Journal, Royal Economic Society, vol. 22(2), pages 188-205.
    17. Steven F. Lehrer & R. Vincent Pohl & Kyungchul Song, 2016. "Targeting Policies: Multiple Testing and Distributional Treatment Effects," NBER Working Papers 22950, National Bureau of Economic Research, Inc.
    18. Goldzahl, Léontine & Hollard, Guillaume & Jusot, Florence, 2018. "Increasing breast-cancer screening uptake: A randomized controlled experiment," Journal of Health Economics, Elsevier, vol. 58(C), pages 228-252.

    More about this item


    Access and download statistics


    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:wly:japmet:v:29:y:2014:i:4:p:612-626. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Wiley Content Delivery). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    We have no references for this item. You can help adding them by using this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.