IDEAS home Printed from https://ideas.repec.org/p/hal/pseptp/halshs-02973595.html
   My bibliography  Save this paper

Estimating the Effect of Treatments Allocated by Randomized Waiting Lists

Author

Listed:
  • Clément Chaisemartin

    (UC Santa Barbara - University of California [Santa Barbara] - UC - University of California)

  • Luc Behaghel

    (PSE - Paris School of Economics - UP1 - Université Paris 1 Panthéon-Sorbonne - ENS-PSL - École normale supérieure - Paris - PSL - Université Paris sciences et lettres - EHESS - École des hautes études en sciences sociales - ENPC - École des Ponts ParisTech - CNRS - Centre National de la Recherche Scientifique - INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement, PJSE - Paris Jourdan Sciences Economiques - UP1 - Université Paris 1 Panthéon-Sorbonne - ENS-PSL - École normale supérieure - Paris - PSL - Université Paris sciences et lettres - EHESS - École des hautes études en sciences sociales - ENPC - École des Ponts ParisTech - CNRS - Centre National de la Recherche Scientifique - INRAE - Institut National de Recherche pour l’Agriculture, l’Alimentation et l’Environnement)

Abstract

Oversubscribed treatments are often allocated using randomized waiting lists. Applicants are ranked randomly, and treatment offers are made following that ranking until all seats are filled. To estimate causal effects, researchers often compare applicants getting and not getting an offer. We show that those two groups are not statistically comparable. Therefore, the estimator arising from that comparison is inconsistent when the number of waitlists goes to infinity. We propose a new estimator, and show that it is consistent, provided the waitlists have at least two seats. Finally, we revisit an application, and we show that using our estimator can lead to a statistically significant difference with respect to the results obtained using the commonly used estimator.

Suggested Citation

  • Clément Chaisemartin & Luc Behaghel, 2020. "Estimating the Effect of Treatments Allocated by Randomized Waiting Lists," PSE-Ecole d'économie de Paris (Postprint) halshs-02973595, HAL.
    • Handle: RePEc:hal:pseptp:halshs-02973595
    DOI: 10.3982/ECTA16032
    as

    Download full text from publisher

    To our knowledge, this item is not available for download. To find whether it is available, there are three options:
    1. Check below whether another version of this item is available online.
    2. Check on the provider's web page whether it is in fact available.
    3. Perform a search for a similarly titled item that would be available.

    Other versions of this item:

    References listed on IDEAS

    as
    1. repec:mpr:mprres:7273 is not listed on IDEAS
    2. Alberto Abadie & Joshua Angrist & Guido Imbens, 2002. "Instrumental Variables Estimates of the Effect of Subsidized Training on the Quantiles of Trainee Earnings," Econometrica, Econometric Society, vol. 70(1), pages 91-117, January.
    3. David J. Deming, 2011. "Better Schools, Less Crime?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(4), pages 2063-2115.
    4. Pablo Ibarraran & Laura Ripani & Bibiana Taboada & Juan Villa & Brigida Garcia, 2014. "Life skills, employability and training for disadvantaged youth: Evidence from a randomized evaluation design," IZA Journal of Labor & Development, Springer;Forschungsinstitut zur Zukunft der Arbeit GmbH (IZA), vol. 3(1), pages 1-24, December.
    5. repec:mpr:mprres:7927 is not listed on IDEAS
    6. Vilsa E. Curto & Roland G. Fryer Jr., 2014. "The Potential of Urban Boarding Schools for the Poor: Evidence from SEED," Journal of Labor Economics, University of Chicago Press, vol. 32(1), pages 65-93.
    7. Will Dobbie & Roland G. Fryer Jr., 2013. "Getting beneath the Veil of Effective Schools: Evidence from New York City," American Economic Journal: Applied Economics, American Economic Association, vol. 5(4), pages 28-60, October.
    8. Clément de Chaisemartin & Luc Behaghel, 2020. "Estimating the Effect of Treatments Allocated by Randomized Waiting Lists," Econometrica, Econometric Society, vol. 88(4), pages 1453-1477, July.
    9. Andrew V. Carter & Kevin T. Schnepel & Douglas G. Steigerwald, 2017. "Asymptotic Behavior of a t -Test Robust to Cluster Heterogeneity," The Review of Economics and Statistics, MIT Press, vol. 99(4), pages 698-709, July.
    10. Justine S. Hastings & Christopher A. Neilson & Seth D. Zimmerman, 2012. "The Effect of School Choice on Intrinsic Motivation and Academic Outcomes," Working Papers 2012-3, Princeton University. Economics Department..
    11. Imbens, Guido W & Angrist, Joshua D, 1994. "Identification and Estimation of Local Average Treatment Effects," Econometrica, Econometric Society, vol. 62(2), pages 467-475, March.
    12. Keisuke Hirano & Guido W. Imbens & Geert Ridder, 2003. "Efficient Estimation of Average Treatment Effects Using the Estimated Propensity Score," Econometrica, Econometric Society, vol. 71(4), pages 1161-1189, July.
    13. David Card & Pablo Ibarrarán & Ferdinando Regalia & David Rosas-Shady & Yuri Soares, 2011. "The Labor Market Impacts of Youth Training in the Dominican Republic," Journal of Labor Economics, University of Chicago Press, vol. 29(2), pages 267-300.
    14. Clément de Chaisemartin, 2017. "Tolerating defiance? Local average treatment effects without monotonicity," Quantitative Economics, Econometric Society, vol. 8(2), pages 367-396, July.
    15. repec:mpr:mprres:7681 is not listed on IDEAS
    16. Luc Behaghel & Clément de Chaisemartin & Marc Gurgand, 2017. "Ready for Boarding? The Effects of a Boarding School for Disadvantaged Students," American Economic Journal: Applied Economics, American Economic Association, vol. 9(1), pages 140-164, January.
    17. Melissa A. Clark & Philip Gleason & Christina Clark Tuttle & Marsha K. Silverberg, 2011. "Do Charter Schools Improve Student Achievement? Evidence from a National Randomized Study," Mathematica Policy Research Reports af41392138504f369930e6f2b, Mathematica Policy Research.
    18. Gut, Allan, 1992. "The weak law of large numbers for arrays," Statistics & Probability Letters, Elsevier, vol. 14(1), pages 49-52, May.
    19. Clément de Chaisemartin & Xavier d'Haultfoeuille, 2012. "Late Again with Defiers," PSE Working Papers halshs-00699646, HAL.
    20. Atila Abdulkadiroğlu & Joshua D. Angrist & Peter D. Hull & Parag A. Pathak, 2016. "Charters without Lotteries: Testing Takeovers in New Orleans and Boston," American Economic Review, American Economic Association, vol. 106(7), pages 1878-1920, July.
    21. Bruno Crépon & Florencia Devoto & Esther Duflo & William Parienté, 2015. "Estimating the Impact of Microcredit on Those Who Take It Up: Evidence from a Randomized Experiment in Morocco," American Economic Journal: Applied Economics, American Economic Association, vol. 7(1), pages 123-150, January.
    22. David J. Deming, 2014. "Using School Choice Lotteries to Test Measures of School Effectiveness," American Economic Review, American Economic Association, vol. 104(5), pages 406-411, May.
    23. Joshua D. Angrist & Peter D. Hull & Parag A. Pathak & Christopher R. Walters, 2017. "Leveraging Lotteries for School Value-Added: Testing and Estimation," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 132(2), pages 871-919.
    24. Sarojini Hirshleifer & David McKenzie & Rita Almeida & Cristobal Ridao‐Cano, 2016. "The Impact of Vocational Training for the Unemployed: Experimental Evidence from Turkey," Economic Journal, Royal Economic Society, vol. 126(597), pages 2115-2146, November.
    25. Blattman, Christopher & Annan, Jeannie, 2016. "Can Employment Reduce Lawlessness and Rebellion? A Field Experiment with High-Risk Men in a Fragile State," American Political Science Review, Cambridge University Press, vol. 110(1), pages 1-17, February.
    26. Atila Abdulkadiroğlu & Joshua D. Angrist & Susan M. Dynarski & Thomas J. Kane & Parag A. Pathak, 2011. "Accountability and Flexibility in Public Schools: Evidence from Boston's Charters And Pilots," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(2), pages 699-748.
    27. Caroline M. Hoxby & Sonali Murarka, 2009. "Charter Schools in New York City: Who Enrolls and How They Affect Their Students' Achievement," NBER Working Papers 14852, National Bureau of Economic Research, Inc.
    28. Christina Clark Tuttle & Philip Gleason & Virginia Knechtel & Ira Nichols-Barrer & Kevin Booker & Gregory Chojnacki & Thomas Coen & Lisbeth Goble, "undated". "Understanding the Effect of KIPP as it Scales: Volume I, Impacts on Achievement and Other Outcomes (Executive Summary)," Mathematica Policy Research Reports 29bd80d4c7bf491f84e9e7d37, Mathematica Policy Research.
    29. Matthew A. Kraft, 2014. "How to Make Additional Time Matter: Integrating Individualized Tutorials into an Extended Day," Education Finance and Policy, MIT Press, vol. 10(1), pages 81-116, November.
    30. repec:mpr:mprres:7680 is not listed on IDEAS
    31. Paloma Acevedo & Guillermo Cruces & Paul Gertler & Sebastian Martinez, 2017. "Living Up to Expectations: How Job Training Made Women Better Off and Men Worse Off," NBER Working Papers 23264, National Bureau of Economic Research, Inc.
    32. Julie Berry Cullen & Brian A Jacob & Steven Levitt, 2006. "The Effect of School Choice on Participants: Evidence from Randomized Lotteries," Econometrica, Econometric Society, vol. 74(5), pages 1191-1230, September.
    33. Tuttle, Christina Clark & Gleason, Philip & Clark, Melissa, 2012. "Using lotteries to evaluate schools of choice: Evidence from a national study of charter schools," Economics of Education Review, Elsevier, vol. 31(2), pages 237-253.
    34. Justine S. Hastings & Thomas J. Kane & Douglas O. Staiger, 2006. "Gender and Performance: Evidence from School Assignment by Randomized Lottery," American Economic Review, American Economic Association, vol. 96(2), pages 232-236, May.
    35. Will Dobbie & Roland G. Fryer, 2011. "Are High-Quality Schools Enough to Increase Achievement among the Poor? Evidence from the Harlem Children's Zone," American Economic Journal: Applied Economics, American Economic Association, vol. 3(3), pages 158-187, July.
    36. Orazio Attanasio & Adriana Kugler & Costas Meghir, 2011. "Subsidizing Vocational Training for Disadvantaged Youth in Colombia: Evidence from a Randomized Trial," American Economic Journal: Applied Economics, American Economic Association, vol. 3(3), pages 188-220, July.
    37. Guido W. Imbens & Donald B. Rubin, 1997. "Estimating Outcome Distributions for Compliers in Instrumental Variables Models," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 64(4), pages 555-574.
    38. Imbens,Guido W. & Rubin,Donald B., 2015. "Causal Inference for Statistics, Social, and Biomedical Sciences," Cambridge Books, Cambridge University Press, number 9780521885881.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Pablo Blanchard & Matías Brum & Paula Carrasco & Cecilia Parada & Ivone Perazzo, 2023. "Employment effects of a social and labor inclusion program," Documentos de Trabajo (working papers) 23-02, Instituto de Economía - IECON.
    2. Clément de Chaisemartin & Luc Behaghel, 2020. "Estimating the Effect of Treatments Allocated by Randomized Waiting Lists," Econometrica, Econometric Society, vol. 88(4), pages 1453-1477, July.
    3. Kirill Borusyak & Peter Hull & Xavier Jaravel, 2023. "Design-Based Identification with Formula Instruments: A Review," NBER Working Papers 31393, National Bureau of Economic Research, Inc.
    4. Joshua D. Angrist, 2022. "Empirical Strategies in Economics: Illuminating the Path From Cause to Effect," Econometrica, Econometric Society, vol. 90(6), pages 2509-2539, November.
    5. Daniel Da Mata & Rodrigo Oliveira & Diana Silva, 2021. "Who benefits from job training programmes?: Evidence from a high-dosage programme in Brazil," WIDER Working Paper Series wp-2021-121, World Institute for Development Economic Research (UNU-WIDER).
    6. Clément de Chaisemartin, 2022. "Trading-off Bias and Variance in Stratified Experiments and in Staggered Adoption Designs, Under a Boundedness Condition on the Magnitude of the Treatment Effect," Working Papers hal-03873919, HAL.
    7. Thomas Le Barbanchon & Diego Ubfal & Federico Araya, 2023. "The Effects of Working While in School: Evidence from Employment Lotteries," American Economic Journal: Applied Economics, American Economic Association, vol. 15(1), pages 383-410, January.
    8. Le Barbanchon, Thomas & Ubfal, Diego & Araya, Federico, 2020. "The Effects of Working While in School: Evidence from Uruguayan Lotteries," IZA Discussion Papers 13929, Institute of Labor Economics (IZA).

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Atila Abdulkadiroğlu & Joshua D. Angrist & Yusuke Narita & Parag A. Pathak, 2017. "Research Design Meets Market Design: Using Centralized Assignment for Impact Evaluation," Econometrica, Econometric Society, vol. 85, pages 1373-1432, September.
    2. Song, Yang, 2019. "Sorting, school performance and quality: Evidence from China," Journal of Comparative Economics, Elsevier, vol. 47(1), pages 238-261.
    3. Figlio, D. & Karbownik, K. & Salvanes, K.G., 2016. "Education Research and Administrative Data," Handbook of the Economics of Education,, Elsevier.
    4. Matthew Davis & Blake Heller, 2019. "No Excuses Charter Schools and College Enrollment: New Evidence from a High School Network in Chicago," Education Finance and Policy, MIT Press, vol. 14(3), pages 414-440, Summer.
    5. Chakravarty, Shubha & Lundberg, Mattias & Nikolov, Plamen & Zenker, Juliane, 2019. "Vocational training programs and youth labor market outcomes: Evidence from Nepal," Journal of Development Economics, Elsevier, vol. 136(C), pages 71-110.
    6. Masi, Barbara, 2018. "A ticket to ride: The unintended consequences of school transport subsidies," Economics of Education Review, Elsevier, vol. 63(C), pages 100-115.
    7. Joshua D. Angrist & Peter D. Hull & Parag A. Pathak & Christopher R. Walters, 2017. "Leveraging Lotteries for School Value-Added: Testing and Estimation," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 132(2), pages 871-919.
    8. Will Dobbie & Roland G. Fryer Jr., 2015. "The Medium-Term Impacts of High-Achieving Charter Schools," Journal of Political Economy, University of Chicago Press, vol. 123(5), pages 985-1037.
    9. Atila Abdulkadiroğlu & Parag A. Pathak & Christopher R. Walters, 2018. "Free to Choose: Can School Choice Reduce Student Achievement?," American Economic Journal: Applied Economics, American Economic Association, vol. 10(1), pages 175-206, January.
    10. Chakravarty, Shubha & Lundberg, Mattias & Nikolov, Plamen & Zenker, Juliane, 2019. "Vocational training programs and youth labor market outcomes: Evidence from Nepal," Journal of Development Economics, Elsevier, vol. 136(C), pages 71-110.
    11. Clément de Chaisemartin, 2022. "Trading-off Bias and Variance in Stratified Experiments and in Staggered Adoption Designs, Under a Boundedness Condition on the Magnitude of the Treatment Effect," Working Papers hal-03873919, HAL.
    12. Joshua D. Angrist & Sarah R. Cohodes & Susan M. Dynarski & Parag A. Pathak & Christopher R. Walters, 2016. "Stand and Deliver: Effects of Boston's Charter High Schools on College Preparation, Entry, and Choice," Journal of Labor Economics, University of Chicago Press, vol. 34(2), pages 275-318.
    13. Peter Hull & Michal Kolesár & Christopher Walters, 2022. "Labor by design: contributions of David Card, Joshua Angrist, and Guido Imbens," Scandinavian Journal of Economics, Wiley Blackwell, vol. 124(3), pages 603-645, July.
    14. Roland G. Fryer, Jr, 2016. "The Production of Human Capital in Developed Countries: Evidence from 196 Randomized Field Experiments," NBER Working Papers 22130, National Bureau of Economic Research, Inc.
    15. Nicholas Bloom & Renata Lemos & Raffaella Sadun & John Van Reenen, 2015. "Does Management Matter in schools?," Economic Journal, Royal Economic Society, vol. 0(584), pages 647-674, May.
    16. Clément de Chaisemartin, 2017. "Tolerating defiance? Local average treatment effects without monotonicity," Quantitative Economics, Econometric Society, vol. 8(2), pages 367-396, July.
    17. Calsamiglia, Caterina & Güell, Maia, 2014. "The Illusion of School Choice: Empirical Evidence from Barcelona," IZA Discussion Papers 8202, Institute of Labor Economics (IZA).
    18. Joe Regan-Stansfield, 2016. "Do good primary schools perform even better as academies?," Working Papers 141167564, Lancaster University Management School, Economics Department.
    19. Joshua D. Angrist, 2022. "Empirical Strategies in Economics: Illuminating the Path From Cause to Effect," Econometrica, Econometric Society, vol. 90(6), pages 2509-2539, November.
    20. Lorenzo Neri & Elisabetta Pasini, 2020. "Heterogeneous Effects of School Autonomy in England," Discussion Paper Series, School of Economics and Finance 202010, School of Economics and Finance, University of St Andrews.

    More about this item

    Keywords

    Waiting lists; Non-takers; Non compliance; Instrumental variable; Local average treatment effect; Randomized controlled trials;
    All these keywords.

    JEL classification:

    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models
    • C26 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Instrumental Variables (IV) Estimation

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:hal:pseptp:halshs-02973595. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Caroline Bauer (email available below). General contact details of provider: https://hal.archives-ouvertes.fr/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.