IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/16935.html
   My bibliography  Save this paper

Inference with Imperfect Randomization: The Case of the Perry Preschool Program

Author

Listed:
  • James J. Heckman
  • Rodrigo Pinto
  • Azeem M. Shaikh
  • Adam Yavitz

Abstract

This paper considers the problem of making inferences about the effects of a program on multiple outcomes when the assignment of treatment status is imperfectly randomized. By imperfect randomization we mean that treatment status is reassigned after an initial randomization on the basis of characteristics that may be observed or unobserved by the analyst. We develop a partial identification approach to this problem that makes use of information limiting the extent to which randomization is imperfect to show that it is still possible to make nontrivial inferences about the effects of the program in such settings. We consider a family of null hypotheses in which each null hypothesis specifies that the program has no effect on one of several outcomes of interest. Under weak assumptions, we construct a procedure for testing this family of null hypotheses in a way that controls the familywise error rate -- the probability of even one false rejection -- in finite samples. We develop our methodology in the context of a reanalysis of the HighScope Perry Preschool program. We find statistically significant effects of the program on a number of different outcomes of interest, including outcomes related to criminal activity for males and females, even after accounting for the imperfectness of the randomization and the multiplicity of null hypotheses.

Suggested Citation

  • James J. Heckman & Rodrigo Pinto & Azeem M. Shaikh & Adam Yavitz, 2011. "Inference with Imperfect Randomization: The Case of the Perry Preschool Program," NBER Working Papers 16935, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:16935
    Note: CH TWP
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w16935.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. James J. Heckman & Ganesh Karapakula, 2019. "The Perry Preschoolers at Late Midlife: A Study in Design-Specific Inference," Working Papers 2019-034, Human Capital and Economic Opportunity Working Group.
    2. Heckman, James J. & Moon, Seong Hyeok & Pinto, Rodrigo & Savelyev, Peter A. & Yavitz, Adam, 2010. "The rate of return to the HighScope Perry Preschool Program," Journal of Public Economics, Elsevier, vol. 94(1-2), pages 114-128, February.
    3. Joseph P. Romano & Michael Wolf, 2005. "Exact and Approximate Stepdown Methods for Multiple Hypothesis Testing," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 94-108, March.
    4. James Heckman & Rodrigo Pinto & Peter Savelyev, 2013. "Understanding the Mechanisms through Which an Influential Early Childhood Program Boosted Adult Outcomes," American Economic Review, American Economic Association, vol. 103(6), pages 2052-2086, October.
    5. Joseph P. Romano & Azeem M. Shaikh & Michael Wolf, 2010. "Hypothesis Testing in Econometrics," Annual Review of Economics, Annual Reviews, vol. 2(1), pages 75-104, September.
    6. James Heckman & Seong Hyeok Moon & Rodrigo Pinto & Peter Savelyev & Adam Yavitz, 2010. "Analyzing social experiments as implemented: A reexamination of the evidence from the HighScope Perry Preschool Program," Quantitative Economics, Econometric Society, vol. 1(1), pages 1-46, July.
    7. Joseph P. Romano & Azeem M. Shaikh, 2010. "Inference for the Identified Set in Partially Identified Econometric Models," Econometrica, Econometric Society, vol. 78(1), pages 169-211, January.
    8. Romano, Joseph P. & Shaikh, Azeem M. & Wolf, Michael, 2008. "Formalized Data Snooping Based On Generalized Error Rates," Econometric Theory, Cambridge University Press, vol. 24(2), pages 404-447, April.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Aviva Aron-Dine & Liran Einav & Amy Finkelstein, 2013. "The RAND Health Insurance Experiment, Three Decades Later," Journal of Economic Perspectives, American Economic Association, vol. 27(1), pages 197-222, Winter.
    2. Arouna, Aminou & Michler, Jeffrey D. & Lokossou, Jourdain C., 2021. "Contract farming and rural transformation: Evidence from a field experiment in Benin," Journal of Development Economics, Elsevier, vol. 151(C).
    3. Daniela Del Boca & Christopher Flinn & Matthew Wiswall, 2016. "Transfers to Households with Children and Child Development," Economic Journal, Royal Economic Society, vol. 126(596), pages 136-183, October.
    4. Sneha Elango & Jorge Luis García & James J. Heckman & Andrés Hojman, 2015. "Early Childhood Education," NBER Chapters, in: Economics of Means-Tested Transfer Programs in the United States, Volume 2, pages 235-297, National Bureau of Economic Research, Inc.
    5. James J. Heckman & Ganesh Karapakula, 2019. "The Perry Preschoolers at Late Midlife: A Study in Design-Specific Inference," Working Papers 2019-034, Human Capital and Economic Opportunity Working Group.
    6. Jeffrey D. Michler & Anna Josephson, 2021. "Recent Developments in Inference: Practicalities for Applied Economics," Papers 2107.09736, arXiv.org.
    7. John A. List & Azeem M. Shaikh & Yang Xu, 2019. "Multiple hypothesis testing in experimental economics," Experimental Economics, Springer;Economic Science Association, vol. 22(4), pages 773-793, December.
    8. Federico A. Bugni & Ivan A. Canay & Azeem M. Shaikh, 2018. "Inference Under Covariate-Adaptive Randomization," Journal of the American Statistical Association, Taylor & Francis Journals, vol. 113(524), pages 1784-1796, October.
    9. Olivier Filatriau & Denis Fougère & Maxime To, 2013. "Will Sooner Be Better ? The Impact of Early Preschool Enrollment on Cognitive and Noncognitive Achievement of Children," Working Papers 2013-10, Center for Research in Economics and Statistics.
    10. Azeem M. Shaikh, 2019. "Inference in Experiments with Matched Pairs," CeMMAP working papers CWP19/19, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    11. Asoni, Andrea, 2011. "Intelligence, Self-confidence and Entrepreneurship," Working Paper Series 887, Research Institute of Industrial Economics.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Doyle, Orla & Harmon, Colm & Heckman, James J. & Logue, Caitriona & Moon, Seong Hyeok, 2017. "Early skill formation and the efficiency of parental investment: A randomized controlled trial of home visiting," Labour Economics, Elsevier, vol. 45(C), pages 40-58.
    2. Ganesh Karapakula & James J. Heckman, 2020. "Using a Satisficing Model of Experimenter Decision-Making to Guide Finite-Sample Inference for Compromised Experiments," Working Papers 2020-063, Human Capital and Economic Opportunity Working Group.
    3. Alexandre Belloni & Victor Chernozhukov & Denis Chetverikov & Christian Hansen & Kengo Kato, 2018. "High-dimensional econometrics and regularized GMM," CeMMAP working papers CWP35/18, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    4. Yann Algan & Elizabeth Beasley & Frank Vitaro & Richard Tremblay, 2014. "The Impact of Non-Cognitive Skills Training on Academic and Non-academic Trajectories: From Childhood to Early Adulthood," Sciences Po publications info:hdl:2441/6s39gt704s9, Sciences Po.
    5. Fabian Kosse & Thomas Deckers & Pia Pinger & Hannah Schildberg-Hörisch & Armin Falk, 2020. "The Formation of Prosociality: Causal Evidence on the Role of Social Environment," Journal of Political Economy, University of Chicago Press, vol. 128(2), pages 434-467.
    6. James J. Heckman & Stefano Mosso, 2014. "The Economics of Human Development and Social Mobility," Annual Review of Economics, Annual Reviews, vol. 6(1), pages 689-733, August.
    7. Raj Chetty & John N. Friedman & Nathaniel Hilger & Emmanuel Saez & Diane Whitmore Schanzenbach & Danny Yagan, 2011. "How Does Your Kindergarten Classroom Affect Your Earnings? Evidence from Project Star," The Quarterly Journal of Economics, Oxford University Press, vol. 126(4), pages 1593-1660.
    8. Doyle, O. & Harmon, C. & Heckman, J.J. & Logue, C,; & Moon, S.H., 2013. "Measuring Investment in Human Capital Formation: An Experimental Analysis of Early Life Outcomes," Health, Econometrics and Data Group (HEDG) Working Papers 13/18, HEDG, c/o Department of Economics, University of York.
    9. Sandner, Malte & Jungmann, Tanja, 2017. "Gender-specific effects of early childhood intervention: Evidence from a randomized controlled trial," Labour Economics, Elsevier, vol. 45(C), pages 59-78.
    10. Orla Doyle, 2017. "The First 2,000 Days and Child Skills: Evidence from a Randomized Experiment of Home Visiting," Working Papers 2017-054, Human Capital and Economic Opportunity Working Group.
    11. Filatriau, Olivier & Fougère, Denis & Tô, Maxime, 2013. "Will Sooner Be Better? The Impact of Early Preschool Enrollment on Cognitive and Noncognitive Achievement of Children," CEPR Discussion Papers 9480, C.E.P.R. Discussion Papers.
    12. Yann Algan & Elizabeth Beasley & Frank Vitaro & Richard E Tremblay, 2014. "The Impact of Non-Cognitive Skills Training on Academic and Non-academic Trajectories: From Childhood to Early Adulthood," Sciences Po publications info:hdl:2441/6s39gt704s9, Sciences Po.
    13. John A. List & Azeem M. Shaikh & Yang Xu, 2019. "Multiple hypothesis testing in experimental economics," Experimental Economics, Springer;Economic Science Association, vol. 22(4), pages 773-793, December.
    14. Peter A. Savelyev & Kegon T. K. Tan, 2019. "Socioemotional Skills, Education, and Health-Related Outcomes of High-Ability Individuals," American Journal of Health Economics, MIT Press, vol. 5(2), pages 250-280, Spring.
    15. Conti, Gabriella & Heckman, James J., 2012. "The Economics of Child Well-Being," IZA Discussion Papers 6930, Institute of Labor Economics (IZA).
    16. Orla Doyle, 2020. "Can Early Intervention have a Sustained Effect on Human Capital?," Working Papers 202001, Geary Institute, University College Dublin.
    17. John List & Azeem Shaikh & Atom Vayalinkal, 2021. "Multiple Testing with Covariate Adjustment in Experimental Economics," Natural Field Experiments 00732, The Field Experiments Website.
    18. Eiji Yamamura, 2017. "Historical education levels and present-day non-cognitive skills," Journal of Economics and Econometrics, Economics and Econometrics Society, vol. 60(1), pages 28-51.
    19. Joseph P. Romano & Azeem M. Shaikh & Michael Wolf, 2010. "Hypothesis Testing in Econometrics," Annual Review of Economics, Annual Reviews, vol. 2(1), pages 75-104, September.
    20. García, Jorge Luis & Heckman, James J. & Ronda, Victor, 2021. "The Lasting Effects of Early Childhood Education on Promoting the Skills and Social Mobility of Disadvantaged African Americans," IZA Discussion Papers 14575, Institute of Labor Economics (IZA).

    More about this item

    JEL classification:

    • C31 - Mathematical and Quantitative Methods - - Multiple or Simultaneous Equation Models; Multiple Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models; Quantile Regressions; Social Interaction Models
    • I21 - Health, Education, and Welfare - - Education - - - Analysis of Education
    • J13 - Labor and Demographic Economics - - Demographic Economics - - - Fertility; Family Planning; Child Care; Children; Youth

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:16935. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.