IDEAS home Printed from https://ideas.repec.org/a/sae/evarev/v41y2017i2p130-154.html
   My bibliography  Save this article

The WWC Attrition Standard

Author

Listed:
  • John Deke
  • Hanley Chiang

Abstract

Background: To limit the influence of attrition bias in assessments of intervention effectiveness, several federal evidence reviews have established a standard for acceptable levels of sample attrition in randomized controlled trials. These evidence reviews include the What Works Clearinghouse (WWC), the Home Visiting Evidence of Effectiveness Review, and the Teen Pregnancy Prevention Evidence Review. We believe the WWC attrition standard may constitute the first use of model-based, empirically supported bounds on attrition bias in the context of a federally sponsored systematic evidence review. Meeting the WWC attrition standard (or one of the attrition standards based on the WWC standard) is now an important consideration for researchers conducting studies that could potentially be reviewed by the WWC (or other evidence reviews). Objectives: The purpose of this article is to explain the WWC attrition model, how that model is used to establish attrition bounds, and to assess the sensitivity of attrition bounds to key parameter values. Research Design: Results are based on equations derived in the article and values generated by applying those equations to a range of parameter values. Results: The authors find that the attrition boundaries are more sensitive to the maximum level of bias that an evidence review is willing to tolerate than to other parameters in the attrition model. Conclusions: The authors conclude that the most productive refinements to existing attrition standards may be with respect to the definition of “maximum tolerable bias.â€

Suggested Citation

  • John Deke & Hanley Chiang, 2017. "The WWC Attrition Standard," Evaluation Review, , vol. 41(2), pages 130-154, April.
    • Handle: RePEc:sae:evarev:v:41:y:2017:i:2:p:130-154
    DOI: 10.1177/0193841X16670047
    as

    Download full text from publisher

    File URL: https://journals.sagepub.com/doi/10.1177/0193841X16670047
    Download Restriction: no
    File URL: https://libkey.io/10.1177/0193841X16670047?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    References listed on IDEAS

    as
    1. Manski, Charles F, 1990. "Nonparametric Bounds on Treatment Effects," American Economic Review, American Economic Association, vol. 80(2), pages 319-323, May.
    2. repec:mpr:mprres:5414 is not listed on IDEAS
    3. repec:mpr:mprres:6174 is not listed on IDEAS
    4. Andrews,Donald W. K. & Stock,James H. (ed.), 2005. "Identification and Inference for Econometric Models," Cambridge Books, Cambridge University Press, number 9780521844413.
    5. repec:mpr:mprres:4871 is not listed on IDEAS
    6. repec:mpr:mprres:6585 is not listed on IDEAS
    7. repec:mpr:mprres:4551 is not listed on IDEAS
    8. Jill Constantine & Daniel Player & Tim Silva & Kristin Hallgren & Mary Grider & John Deke, 2009. "An Evaluation of Teachers Trained Through Different Routes to Certification," Mathematica Policy Research Reports 1a9e2910a01843c2babbc8c64, Mathematica Policy Research.
    9. Jill Constantine, 2009. "Evaluation of Teachers Trained Through Different Routes to Certification," Mathematica Policy Research Reports 6e759a791c77408a816add035, Mathematica Policy Research.
    10. Susanne James-Burdumy & Mark Dynarski & Mary Moore & John Deke & Wendy Mansfield & Carol Pistorino, "undated". "When Schools Stay Open Late: The National Evaluation of the 21st-Century Community Learning Centers Program," Mathematica Policy Research Reports 86c8d763ea6c4acebca8464c5, Mathematica Policy Research.
    11. David Greenberg & Burt S. Barnow, 2014. "Flaws in Evaluations of Social Programs," Evaluation Review, , vol. 38(5), pages 359-387, October.
    12. repec:mpr:mprres:6175 is not listed on IDEAS
    13. repec:mpr:mprres:7338 is not listed on IDEAS
    14. Susanne James-Burdumy & Mark Dynarski & John Deke, 2005. "When Elementary Schools Stay Open Late: Results from The National Evaluation of the 21st-Century Community Learning Centers Program," Mathematica Policy Research Reports 747640229207407f9f0f09abf, Mathematica Policy Research.
    15. repec:mpr:mprres:5863 is not listed on IDEAS
    16. Neil Seftor, "undated". "Raising the Bar," Mathematica Policy Research Reports 135e82100e784dad803fe9c89, Mathematica Policy Research.
    17. David S. Lee, 2009. "Training, Wages, and Sample Selection: Estimating Sharp Bounds on Treatment Effects," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 76(3), pages 1071-1102.
    18. Peter Z. Schochet, "undated". "Statistical Power for Random Assignment Evaluations of Education Programs," Mathematica Policy Research Reports 6749d31ad72d4acf988f7dce5, Mathematica Policy Research.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Ben Weidmann & Luke Miratrix, 2021. "Missing, presumed different: Quantifying the risk of attrition bias in education evaluations," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 184(2), pages 732-760, April.
    2. Jaime Thomas & Sarah A. Avellar & John Deke & Philip Gleason, 2017. "Matched Comparison Group Design Standards in Systematic Reviews of Early Childhood Interventions," Evaluation Review, , vol. 41(3), pages 240-279, June.
    3. Douglas J. Besharov & Douglas M. Call & Jason M. Scott, 2020. "PROTOCOL: Early childhood education programs for improving the development and achievement of low‐income children: a systematic review," Campbell Systematic Reviews, John Wiley & Sons, vol. 16(3), September.
    4. Jacob Alex Klerman, 2017. "Special Issue Editor’s Overview Essay," Evaluation Review, , vol. 41(3), pages 175-182, June.
    5. Richard Hendra & Aaron Hill, 2019. "Rethinking Response Rates: New Evidence of Little Relationship Between Survey Response Rates and Nonresponse Bias," Evaluation Review, , vol. 43(5), pages 307-330, October.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Lisa Dragoset & Jaime Thomas & Mariesa Herrmann & John Deke & Susanne James-Burdumy & Cheryl Graczewski & Andrea Boyle & Rachel Upton & Courtney Tanenbaum & Jessica Giffin, "undated". "School Improvement Grants: Implementation and Effectiveness (Final Report)," Mathematica Policy Research Reports 76bce3f4bb0944f29a481fae0, Mathematica Policy Research.
    2. Lisa Dragoset & Susanne James-Burdumy & Kristin Hallgren & Irma Perez-Johnson & Mariesa Herrmann & Christina Tuttle & Megan Hague Angus & Rebecca Herman & Matthew Murray & Courtney Tanenbaum & Cheryl , 2015. "Usage of Practices Promoted by School Improvement Grants," Mathematica Policy Research Reports 8e99f01663504ef5b9f8357f6, Mathematica Policy Research.
    3. Peter Z. Schochet & Hanley Chiang, "undated". "Technical Methods Report: Estimation and Identification of the Complier Average Causal Effect Parameter in Education RCTs," Mathematica Policy Research Reports 947d1823e3ff42208532a763d, Mathematica Policy Research.
    4. John Deke, "undated". "Causal Validity Considerations for Including High Quality Non-Experimental Evidence in Systematic Reviews," Mathematica Policy Research Reports 676a04feb19e4904a052ba2e7, Mathematica Policy Research.
    5. Kaitlin Anderson & Gema Zamarro & Jennifer Steele & Trey Miller, 2021. "Comparing Performance of Methods to Deal With Differential Attrition in Randomized Experimental Evaluations," Evaluation Review, , vol. 45(1-2), pages 70-104, February.
    6. John Deke & Thomas Wei & Tim Kautz, "undated". "Asymdystopia: The Threat of Small Biases in Evaluations of Education Interventions that Need to be Powered to Detect Small Impacts," Mathematica Policy Research Reports f0ff8f86e3c34dc8baaf22b56, Mathematica Policy Research.
    7. Turner, Alex J. & Fichera, Eleonora & Sutton, Matt, 2021. "The effects of in-utero exposure to influenza on mental health and mortality risk throughout the life-course," Economics & Human Biology, Elsevier, vol. 43(C).
    8. Kjetil Bjorvatn & Alexander W. Cappelen & Linda Helgesson Sekei & Erik Ø. Sørensen & Bertil Tungodden, 2020. "Teaching Through Television: Experimental Evidence on Entrepreneurship Education in Tanzania," Management Science, INFORMS, vol. 66(6), pages 2308-2325, June.
    9. Xintong Wang & Carlos A. Flores & Alfonso Flores-Lagunes, 2020. "The Effects of Vietnam-Era Military Service on the Long-Term Health of Veterans: A Bounds Analysis," Center for Policy Research Working Papers 234, Center for Policy Research, Maxwell School, Syracuse University.
    10. McGovern, Mark E. & Canning, David & Bärnighausen, Till, 2018. "Accounting for non-response bias using participation incentives and survey design: An application using gift vouchers," Economics Letters, Elsevier, vol. 171(C), pages 239-244.
    11. Vegas, E & Ganimian, A. J., 2013. "Theory and Evidence on Teacher Policies in Developed and Developing Countries," Working Paper 104291, Harvard University OpenScholar.
    12. repec:mpr:mprres:6286 is not listed on IDEAS
    13. Guido W. Imbens & Jeffrey M. Wooldridge, 2009. "Recent Developments in the Econometrics of Program Evaluation," Journal of Economic Literature, American Economic Association, vol. 47(1), pages 5-86, March.
    14. Firpo, Sergio Pinheiro & Pinto, Rafael de Carvalho Cayres, 2012. "Combining Strategies for the Estimation of Treatment Effects," Brazilian Review of Econometrics, Sociedade Brasileira de Econometria - SBE, vol. 32(1), March.
    15. Semenova, Vira, 2023. "Debiased machine learning of set-identified linear models," Journal of Econometrics, Elsevier, vol. 235(2), pages 1725-1746.
    16. Hanley S. Chiang & Melissa A. Clark & Sheena McConnell, "undated". "Supplying Disadvantaged Schools with Effective Teachers: Experimental Evidence on Secondary Math Teachers from Teach For America," Mathematica Policy Research Reports 4623b3b72e154403be21e3878, Mathematica Policy Research.
    17. Matthew A. Kraft, 2014. "How to Make Additional Time Matter: Integrating Individualized Tutorials into an Extended Day," Education Finance and Policy, MIT Press, vol. 10(1), pages 81-116, November.
    18. Aizawa, T.;, 2019. "Reviewing the Existing Evidence of the Conditional Cash Transfer in India through the Partial Identification Approach," Health, Econometrics and Data Group (HEDG) Working Papers 19/24, HEDG, c/o Department of Economics, University of York.
    19. Justus J. Randolph & Anaya Bryson & Lakshmi Menon & Stephen Michaels & Debra Leigh Walls Rosenstein & Warren McPherson, 2016. "PROTOCOL: Montessori Education for Improving Academic and Social/Behavioral Outcomes for Elementary Students," Campbell Systematic Reviews, John Wiley & Sons, vol. 12(1), pages 1-32.
    20. Roland G. Fryer, Jr, 2016. "The Production of Human Capital in Developed Countries: Evidence from 196 Randomized Field Experiments," NBER Working Papers 22130, National Bureau of Economic Research, Inc.
    21. Flores, Carlos A. & Flores-Lagunes, Alfonso, 2009. "Identification and Estimation of Causal Mechanisms and Net Effects of a Treatment under Unconfoundedness," IZA Discussion Papers 4237, Institute of Labor Economics (IZA).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:sae:evarev:v:41:y:2017:i:2:p:130-154. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: SAGE Publications (email available below). General contact details of provider: .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.