IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp16675.html
   My bibliography  Save this paper

Dealing With Imperfect Randomization: Inference for the Highscope Perry Preschool Program

Author

Listed:
  • Heckman, James J.

    (University of Chicago)

  • Pinto, Rodrigo

    (University of California, Los Angeles)

  • Shaikh, Azeem M.

    (University of Chicago)

Abstract

This paper considers the problem of making inferences about the effects of a program on multiple outcomes when the assignment of treatment status is imperfectly randomized. By imperfect randomization we mean that treatment status is reassigned after an initial randomization on the basis of characteristics that may be observed or unobserved by the analyst. We develop a partial identification approach to this problem that makes use of information limiting the extent to which randomization is imperfect to show that it is still possible to make nontrivial inferences about the effects of the program in such settings. We consider a family of null hypotheses in which each null hypothesis species that the program has no effect on one of many outcomes of interest. Under weak assumptions, we construct a procedure for testing this family of null hypotheses in a way that controls the familywise error rate – the probability of even one false rejection – in finite samples. We develop our methodology in the context of a reanalysis of the HighScope Perry Preschool program. We find statistically signicant effects of the program on a number of different outcomes of interest, including outcomes related to criminal activity for males and females, even after accounting for imperfections in the randomization and the multiplicity of null hypotheses.

Suggested Citation

  • Heckman, James J. & Pinto, Rodrigo & Shaikh, Azeem M., 2023. "Dealing With Imperfect Randomization: Inference for the Highscope Perry Preschool Program," IZA Discussion Papers 16675, Institute of Labor Economics (IZA).
    • Handle: RePEc:iza:izadps:dp16675
    as

    Download full text from publisher

    File URL: https://docs.iza.org/dp16675.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Azeem M. Shaikh & Panos Toulis, 2021. "Randomization Tests in Observational Studies With Staggered Adoption of Treatment," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 116(536), pages 1835-1848, October.
    2. Ivan A. Canay & Vishal Kamat, 2017. "Approximate permutation tests and induced order statistics in the regression discontinuity design," CeMMAP working papers 21/17, Institute for Fiscal Studies.
    3. Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2018. "Inference Under Covariate-Adaptive Randomization," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(524), pages 1784-1796, October.
    4. G W Basse & A Feller & P Toulis, 2019. "Randomization tests of causal effects under interference," Biometrika, Biometrika Trust, vol. 106(2), pages 487-494.
    5. Ivan A Canay & Vishal Kamat, 2018. "Approximate Permutation Tests and Induced Order Statistics in the Regression Discontinuity Design," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 85(3), pages 1577-1608.
    6. Joseph P. Romano & Azeem M. Shaikh, 2010. "Inference for the Identified Set in Partially Identified Econometric Models," Econometrica, Econometric Society, vol. 78(1), pages 169-211, January.
    7. James Heckman & Rodrigo Pinto & Peter Savelyev, 2013. "Understanding the Mechanisms through Which an Influential Early Childhood Program Boosted Adult Outcomes," American Economic Review, American Economic Association, vol. 103(6), pages 2052-2086, October.
    8. Romano, Joseph P. & Shaikh, Azeem M. & Wolf, Michael, 2008. "Formalized Data Snooping Based On Generalized Error Rates," Econometric Theory, Cambridge University Press, vol. 24(2), pages 404-447, April.
    9. Peter Ganong & Simon Jäger, 2018. "A Permutation Test for the Regression Kink Design," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(522), pages 494-504, April.
    10. Jorge Luis García & James J. Heckman & Victor Ronda, 2023. "The Lasting Effects of Early-Childhood Education on Promoting the Skills and Social Mobility of Disadvantaged African Americans and Their Children," Journal of Political Economy, University of Chicago Press, vol. 131(6), pages 1477-1506.
    11. Joseph P. Romano & Michael Wolf, 2005. "Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 94-108, March.
    12. Ivan A. Canay & Andres Santos & Azeem M. Shaikh, 2021. "The Wild Bootstrap with a “Small†Number of “Large†Clusters," The Review of Economics and Statistics, MIT Press, vol. 103(2), pages 346-363, May.
    13. Cai Yong & Canay Ivan A. & Kim Deborah & Shaikh Azeem M., 2023. "On the Implementation of Approximate Randomization Tests in Linear Models with a Small Number of Clusters," Journal of Econometric Methods, De Gruyter, vol. 12(1), pages 85-103, January.
    14. James J. Heckman & Ganesh Karapakula, 2019. "The Perry Preschoolers at Late Midlife: A Study in Design-Specific Inference," Working Papers 2019-034, Human Capital and Economic Opportunity Working Group.
    15. Ivan A. Canay & Joseph P. Romano & Azeem M. Shaikh, 2017. "Randomization Tests Under an Approximate Symmetry Assumption," Econometrica, Econometric Society, vol. 85, pages 1013-1030, May.
    16. Joseph P. Romano & Azeem M. Shaikh & Michael Wolf, 2010. "Hypothesis Testing in Econometrics," Annual Review of Economics, Annual Reviews, vol. 2(1), pages 75-104, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Rodrigo Pinto & Azeem Shaikh & Adam Yavitz & James Heckman, 2010. "Inference with Imperfect Randomization: The Case of the Perry Preschool Program," 2010 Meeting Papers 1336, Society for Economic Dynamics.
    2. John A. List & Azeem M. Shaikh & Atom Vayalinkal, 2023. "Multiple testing with covariate adjustment in experimental economics," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 38(6), pages 920-939, September.
    3. Ivan A Canay & Vishal Kamat, 2018. "Approximate Permutation Tests and Induced Order Statistics in the Regression Discontinuity Design," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 85(3), pages 1577-1608.
    4. Purevdorj Tuvaandorj, 2021. "Robust Permutation Tests in Linear Instrumental Variables Regression," Papers 2111.13774, arXiv.org, revised Jun 2023.
    5. John A. List & Azeem M. Shaikh & Yang Xu, 2019. "Multiple hypothesis testing in experimental economics," Experimental Economics, Springer;Economic Science Association, vol. 22(4), pages 773-793, December.
    6. Joseph P. Romano & Azeem M. Shaikh & Michael Wolf, 2010. "Hypothesis Testing in Econometrics," Annual Review of Economics, Annual Reviews, vol. 2(1), pages 75-104, September.
    7. MacKinnon, James G. & Nielsen, Morten Ørregaard & Webb, Matthew D., 2023. "Cluster-robust inference: A guide to empirical practice," Journal of Econometrics, Elsevier, vol. 232(2), pages 272-299.
    8. James J Heckman & Ganesh Karapakula, 2021. "Using a satisficing model of experimenter decision-making to guide finite-sample inference for compromised experiments," The Econometrics Journal, Royal Economic Society, vol. 24(2), pages 1-39.
    9. García, Jorge Luis & Heckman, James J. & Ronda, Victor, 2021. "The Lasting Effects of Early Childhood Education on Promoting the Skills and Social Mobility of Disadvantaged African Americans," IZA Discussion Papers 14575, Institute of Labor Economics (IZA).
    10. Zhao, Anqi & Ding, Peng, 2021. "Covariate-adjusted Fisher randomization tests for the average treatment effect," Journal of Econometrics, Elsevier, vol. 225(2), pages 278-294.
    11. Alexandre Belloni & Victor Chernozhukov & Denis Chetverikov & Christian Hansen & Kengo Kato, 2018. "High-Dimensional Econometrics and Regularized GMM," Papers 1806.01888, arXiv.org, revised Jun 2018.
    12. Doyle, Orla & Harmon, Colm & Heckman, James J. & Logue, Caitriona & Moon, Seong Hyeok, 2017. "Early skill formation and the efficiency of parental investment: A randomized controlled trial of home visiting," Labour Economics, Elsevier, vol. 45(C), pages 40-58.
    13. Zeng-Hua Lu, 2019. "Extended MinP Tests of Multiple Hypotheses," Papers 1911.04696, arXiv.org.
    14. Simon Heß, 2017. "Randomization inference with Stata: A guide and software," Stata Journal, StataCorp LP, vol. 17(3), pages 630-651, September.
    15. Fabian Kosse & Thomas Deckers & Pia Pinger & Hannah Schildberg-Hörisch & Armin Falk, 2020. "The Formation of Prosociality: Causal Evidence on the Role of Social Environment," Journal of Political Economy, University of Chicago Press, vol. 128(2), pages 434-467.
    16. Berger, Eva M. & Fehr, Ernst & Hermes, Henning & Schunk, Daniel & Winkel, Kirsten, 2020. "The Impact of Working Memory Training on Children's Cognitive and Noncognitive Skills," IZA Discussion Papers 13338, Institute of Labor Economics (IZA).
    17. Agostinelli, Francesco & Avitabile, Ciro & Bobba, Matteo, 2021. "Enhancing Human Capital in Children: A Case Study on Scaling," TSE Working Papers 21-1196, Toulouse School of Economics (TSE), revised Oct 2023.
    18. Doyle, O. & Harmon, C. & Heckman, J.J. & Logue, C,; & Moon, S.H., 2013. "Measuring Investment in Human Capital Formation: An Experimental Analysis of Early Life Outcomes," Health, Econometrics and Data Group (HEDG) Working Papers 13/18, HEDG, c/o Department of Economics, University of York.
    19. Michael P. Leung, 2023. "Network Cluster‐Robust Inference," Econometrica, Econometric Society, vol. 91(2), pages 641-667, March.
    20. Ann P. Bartel & Maya Rossin-Slater & Christopher J. Ruhm & Meredith Slopen & Jane Waldfogel, 2021. "The Impact of Paid Family Leave on Employers: Evidence from New York," NBER Working Papers 28672, National Bureau of Economic Research, Inc.

    More about this item

    Keywords

    exact inference; experiments; familywise error rate; imperfect randomization; multiple testing; multiple outcomes; permutation tests; Perry Preschool Program; program evaluation;
    All these keywords.

    JEL classification:

    • C31 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models; Quantile Regressions; Social Interaction Models
    • I21 - Health, Education, and Welfare - - Education - - - Analysis of Education
    • J13 - Labor and Demographic Economics - - Demographic Economics - - - Fertility; Family Planning; Child Care; Children; Youth

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp16675. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Holger Hinte (email available below). General contact details of provider: https://edirc.repec.org/data/izaaade.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.