IDEAS home Printed from https://ideas.repec.org/a/bla/jorssa/v184y2021i2p732-760.html
   My bibliography  Save this article

Missing, presumed different: Quantifying the risk of attrition bias in education evaluations

Author

Listed:
  • Ben Weidmann
  • Luke Miratrix

Abstract

We estimate the magnitude of attrition bias for 10 randomized controlled trials (RCTs) in education. We make use of a unique feature of administrative school data in England that allows us to analyse post‐test academic outcomes for nearly all students, including those who originally dropped out of the RCTs we analyse. We find that the typical magnitude of attrition bias is 0.015 effect size units (ES), with no estimate greater than 0.034 ES. This suggests that, in practice, the risk of attrition bias is limited. However, this risk should not be ignored as we find some evidence against the common ‘Missing At Random’ assumption. Attrition appears to be more problematic for treated units. We recommend that researchers incorporate uncertainty due to attrition bias, as well as performing sensitivity analyses based on the types of attrition mechanisms that are observed in practice.

Suggested Citation

  • Ben Weidmann & Luke Miratrix, 2021. "Missing, presumed different: Quantifying the risk of attrition bias in education evaluations," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(2), pages 732-760, April.
    • Handle: RePEc:bla:jorssa:v:184:y:2021:i:2:p:732-760
    DOI: 10.1111/rssa.12677
    as

    Download full text from publisher

    File URL: https://doi.org/10.1111/rssa.12677
    Download Restriction: no
    File URL: https://libkey.io/10.1111/rssa.12677?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Manski, Charles F, 1990. "Nonparametric Bounds on Treatment Effects," American Economic Review, American Economic Association, vol. 80(2), pages 319-323, May.
    2. Dalia Ghanem & Sarojini Hirshleifer & Karen Ortiz-Becerra, 2019. "Testing Attrition Bias in Field Experiments," Working Papers 202218, University of California at Riverside, Department of Economics, revised Oct 2022.
    3. David Greenberg & Burt S. Barnow, 2014. "Flaws in Evaluations of Social Programs," Evaluation Review, , vol. 38(5), pages 359-387, October.
    4. Dalia Ghanem & Sarojini Hirshleifer & Karen Ortiz-Becerra, 2019. "Testing for Attrition Bias in Field Experiments," Working Papers 202010, University of California at Riverside, Department of Economics, revised Mar 2020.
    5. Harvey Goldstein & James R. Carpenter & William J. Browne, 2014. "Fitting multilevel multivariate models with missing data in responses and covariates that may include interactions and non-linear terms," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 177(2), pages 553-564, February.
    6. Rolf Sundberg, 2003. "Conditional statistical inference and quantification of relevance," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 65(1), pages 299-315, February.
    7. John Deke & Hanley Chiang, 2017. "The WWC Attrition Standard," Evaluation Review, , vol. 41(2), pages 130-154, April.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Hardy, Morgan & Kim, Seongyoon & McCasland, Jamie & Menzel, Andreas & Witte, Marc, 2024. "Allocating labor across small firms: Experimental evidence on information constraints," Journal of Development Economics, Elsevier, vol. 171(C).
    2. John Deke & Hanley Chiang, 2017. "The WWC Attrition Standard," Evaluation Review, , vol. 41(2), pages 130-154, April.
    3. Annie Alcid & Erwin Bulte & Robert Lensink & Aussi Sayinzoga & Mark Treurniet, 2023. "Short- and Medium-term Impacts of Employability Training: Evidence from a Randomised Field Experiment in Rwanda," Journal of African Economies, Centre for the Study of African Economies, vol. 32(3), pages 296-328.
    4. Rafkin, Charlie & Shreekumar, Advik & Vautrey, Pierre-Luc, 2021. "When guidance changes: Government stances and public beliefs," Journal of Public Economics, Elsevier, vol. 196(C).
    5. Fulya Ersoy, 2021. "Returns to effort: experimental evidence from an online language platform," Experimental Economics, Springer;Economic Science Association, vol. 24(3), pages 1047-1073, September.
    6. Guigonan S. Adjognon & Daan van Soest & Jonas Guthoff, 2021. "Reducing Hunger with Payments for Environmental Services (PES): Experimental Evidence from Burkina Faso," American Journal of Agricultural Economics, John Wiley & Sons, vol. 103(3), pages 831-857, May.
    7. Tarek Azzam & Michael Bates & David Fairris, 2019. "Do Learning Communities Increase First Year College Retention? Testing Sample Selection and External Validity of Randomized Control Trials," Working Papers 202002, University of California at Riverside, Department of Economics.
    8. Kaitlin Anderson & Gema Zamarro & Jennifer Steele & Trey Miller, 2021. "Comparing Performance of Methods to Deal With Differential Attrition in Randomized Experimental Evaluations," Evaluation Review, , vol. 45(1-2), pages 70-104, February.
    9. Victor Aguirregabiria, 2006. "Another Look at the Identification of Dynamic Discrete Decision Processes: With an Application to Retirement Behavior," 2006 Meeting Papers 169, Society for Economic Dynamics.
    10. C, Loran & Eckbo, Espen & Lu, Ching-Chih, 2014. "Does Executive Compensation Reflect Default Risk?," UiS Working Papers in Economics and Finance 2014/11, University of Stavanger.
    11. repec:pri:cepsud:109rothstein is not listed on IDEAS
    12. Turner, Alex J. & Fichera, Eleonora & Sutton, Matt, 2021. "The effects of in-utero exposure to influenza on mental health and mortality risk throughout the life-course," Economics & Human Biology, Elsevier, vol. 43(C).
    13. Brock, William A. & Durlauf, Steven N., 2007. "Identification of binary choice models with social interactions," Journal of Econometrics, Elsevier, vol. 140(1), pages 52-75, September.
    14. Patrick Kline & Christopher R. Walters, 2019. "On Heckits, LATE, and Numerical Equivalence," Econometrica, Econometric Society, vol. 87(2), pages 677-696, March.
    15. Kjetil Bjorvatn & Alexander W. Cappelen & Linda Helgesson Sekei & Erik Ø. Sørensen & Bertil Tungodden, 2020. "Teaching Through Television: Experimental Evidence on Entrepreneurship Education in Tanzania," Management Science, INFORMS, vol. 66(6), pages 2308-2325, June.
    16. Card, David & Rothstein, Jesse, 2007. "Racial segregation and the black-white test score gap," Journal of Public Economics, Elsevier, vol. 91(11-12), pages 2158-2184, December.
    17. Gu, Jiaying & Russell, Thomas M., 2023. "Partial identification in nonseparable binary response models with endogenous regressors," Journal of Econometrics, Elsevier, vol. 235(2), pages 528-562.
    18. Armstrong, Timothy B., 2014. "Weighted KS statistics for inference on conditional moment inequalities," Journal of Econometrics, Elsevier, vol. 181(2), pages 92-116.
    19. David S. Lee, 2002. "Trimming for Bounds on Treatment Effects with Missing Outcomes," NBER Technical Working Papers 0277, National Bureau of Economic Research, Inc.
    20. Victor Chernozhukov & Sokbae Lee & Adam M. Rosen, 2013. "Intersection Bounds: Estimation and Inference," Econometrica, Econometric Society, vol. 81(2), pages 667-737, March.
    21. John Engberg & Dennis Epple & Jason Imbrogno & Holger Sieg & Ron Zimmer, 2014. "Evaluating Education Programs That Have Lotteried Admission and Selective Attrition," Journal of Labor Economics, University of Chicago Press, vol. 32(1), pages 27-63.

    More about this item

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:bla:jorssa:v:184:y:2021:i:2:p:732-760. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Wiley Content Delivery (email available below). General contact details of provider: https://edirc.repec.org/data/rssssea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.