IDEAS home Printed from https://ideas.repec.org/p/mpr/mprres/f0ff8f86e3c34dc8baaf22b564ed8081.html

Asymdystopia: The Threat of Small Biases in Evaluations of Education Interventions that Need to be Powered to Detect Small Impacts

Author

Listed:
  • John Deke
  • Thomas Wei
  • Tim Kautz

Abstract

The authors examine the potential for small biases to increase the risk of making false inferences as studies are powered to detect smaller impacts and recommend strategies researchers can use to avoid or mitigate these biases.

Suggested Citation

  • John Deke & Thomas Wei & Tim Kautz, "undated". "Asymdystopia: The Threat of Small Biases in Evaluations of Education Interventions that Need to be Powered to Detect Small Impacts," Mathematica Policy Research Reports f0ff8f86e3c34dc8baaf22b56, Mathematica Policy Research.
    • Handle: RePEc:mpr:mprres:f0ff8f86e3c34dc8baaf22b564ed8081
    as

    Download full text from publisher

    File URL: https://www.mathematica.org/-/media/publications/pdfs/education/2017/asymdystopia-ies-rpt.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Jeffrey Max & Jill Constantine & Alison Wellington & Kristin Hallgren & Steven Glazerman & Hanley Chiang & Cecilia Speroni, "undated". "Evaluation of the Teacher Incentive Fund: Implementation and Early Impacts of Pay-for-Performance After One Year: Executive Summary," Mathematica Policy Research Reports d57be8f9aed44f0ca1bc582af, Mathematica Policy Research.
    2. repec:mpr:mprres:6372 is not listed on IDEAS
    3. Sebastian Calonico & Matias D. Cattaneo & Rocio Titiunik, 2014. "Robust Nonparametric Confidence Intervals for Regression‐Discontinuity Designs," Econometrica, Econometric Society, vol. 82, pages 2295-2326, November.
    4. Larissa Campuzano & Mark Dynarski & Roberto Agodini & Kristina Rall, "undated". "Effectiveness of Reading and Mathematics Software Products: Findings from Two Student Cohorts," Mathematica Policy Research Reports c6255408bda44f82910991c48, Mathematica Policy Research.
    5. Susanne James-Burdumy & John Deke & Julieta Lugo-Gil & Nancy Carey & Alan Hershey & Russell Gersten & Rebecca Newman-Gonchar & Joseph Dimino & Kelly Haymond & Bonnie Faddis, "undated". "Effectiveness of Selected Supplemental Reading Comprehension Interventions: Findings from Two Student Cohorts," Mathematica Policy Research Reports 9bd673f8e8f9420787c84c985, Mathematica Policy Research.
    6. repec:mpr:mprres:6174 is not listed on IDEAS
    7. John Deke & Lisa Dragoset, "undated". "Statistical Power for Regression Discontinuity Designs in Education: Empirical Estimates of Design Effects Relative to Randomized Controlled Trials," Mathematica Policy Research Reports a4f1d03eb7bf427a8983d4736, Mathematica Policy Research.
    8. repec:mpr:mprres:7346 is not listed on IDEAS
    9. Peter Z. Schochet, "undated". "Technical Methods Report: Statistical Power for Regression Discontinuity Designs in Education Evaluations," Mathematica Policy Research Reports 61fb6c057561451a8a6074508, Mathematica Policy Research.
    10. repec:mpr:mprres:6585 is not listed on IDEAS
    11. repec:mpr:mprres:7339 is not listed on IDEAS
    12. Jill Constantine & Daniel Player & Tim Silva & Kristin Hallgren & Mary Grider & John Deke, 2009. "An Evaluation of Teachers Trained Through Different Routes to Certification," Mathematica Policy Research Reports 1a9e2910a01843c2babbc8c64, Mathematica Policy Research.
    13. Jill Constantine, 2009. "Evaluation of Teachers Trained Through Different Routes to Certification," Mathematica Policy Research Reports 6e759a791c77408a816add035, Mathematica Policy Research.
    14. David Greenberg & Burt S. Barnow, 2014. "Flaws in Evaluations of Social Programs," Evaluation Review, , vol. 38(5), pages 359-387, October.
    15. repec:mpr:mprres:6175 is not listed on IDEAS
    16. repec:mpr:mprres:7338 is not listed on IDEAS
    17. repec:mpr:mprres:6693 is not listed on IDEAS
    18. Susanne James‐Burdumy & Mark Dynarski & John Deke, 2008. "After‐School Program Effects On Behavior: Results From The 21st Century Community Learning Centers Program National Evaluation," Economic Inquiry, Western Economic Association International, vol. 46(1), pages 13-18, January.
    19. Edward E. Leamer, 2010. "Tantalus on the Road to Asymptopia," Journal of Economic Perspectives, American Economic Association, vol. 24(2), pages 31-46, Spring.
    20. repec:mpr:mprres:5879 is not listed on IDEAS
    21. Guido Imbens & Karthik Kalyanaraman, 2012. "Optimal Bandwidth Choice for the Regression Discontinuity Estimator," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 79(3), pages 933-959.
    22. Hanley Chiang & Alison Wellington & Kristin Hallgren & Cecilia Speroni & Mariesa Herrmann & Steven Glazerman & Jill Constantine, "undated". "Evaluation of the Teacher Incentive Fund: Implementation and Impacts of Pay-for-Performance After Two Years," Mathematica Policy Research Reports 4f123c78ddb644079b88d2bb4, Mathematica Policy Research.
    23. repec:mpr:mprres:6181 is not listed on IDEAS
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Stanley Pogrow, 2019. "How Effect Size (Practical Significance) Misleads Clinical Practice: The Case for Switching to Practical Benefit to Assess Applied Research Findings," The American Statistician, Taylor & Francis Journals, vol. 73(S1), pages 223-234, March.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. John Deke & Lisa Dragoset, "undated". "Statistical Power for Regression Discontinuity Designs in Education: Empirical Estimates of Design Effects Relative to Randomized Controlled Trials," Mathematica Policy Research Reports a4f1d03eb7bf427a8983d4736, Mathematica Policy Research.
    2. Lisa Dragoset & Jaime Thomas & Mariesa Herrmann & John Deke & Susanne James-Burdumy & Cheryl Graczewski & Andrea Boyle & Rachel Upton & Courtney Tanenbaum & Jessica Giffin, "undated". "School Improvement Grants: Implementation and Effectiveness (Final Report)," Mathematica Policy Research Reports 76bce3f4bb0944f29a481fae0, Mathematica Policy Research.
    3. John Deke & Hanley Chiang, 2017. "The WWC Attrition Standard," Evaluation Review, , vol. 41(2), pages 130-154, April.
    4. repec:mpr:mprres:7486 is not listed on IDEAS
    5. Rahul Singh & Moses Stewart, 2025. "Placebo Discontinuity Design," Papers 2507.12693, arXiv.org.
    6. Chen, Yi & Zhao, Yi, 2022. "The timing of first marriage and subsequent life outcomes: Evidence from a natural experiment," Journal of Comparative Economics, Elsevier, vol. 50(3), pages 713-731.
    7. Ivan A Canay & Vishal Kamat, 2018. "Approximate Permutation Tests and Induced Order Statistics in the Regression Discontinuity Design," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 85(3), pages 1577-1608.
    8. Toro, Weily & Tigre, Robson & Sampaio, Breno, 2015. "Daylight Saving Time and incidence of myocardial infarction: Evidence from a regression discontinuity design," Economics Letters, Elsevier, vol. 136(C), pages 1-4.
    9. Cl'ement de Chaisemartin & Diego Ciccia & Xavier D'Haultf{oe}uille & Felix Knau, 2024. "Difference-in-Differences Estimators When No Unit Remains Untreated," Papers 2405.04465, arXiv.org, revised Aug 2025.
    10. Shaun M. Dougherty, 2018. "The Effect of Career and Technical Education on Human Capital Accumulation: Causal Evidence from Massachusetts," Education Finance and Policy, MIT Press, vol. 13(2), pages 119-148, Spring.
    11. Christopher S. Carpenter & Carlos Dobkin & Casey Warman, 2016. "The Mechanisms of Alcohol Control," Journal of Human Resources, University of Wisconsin Press, vol. 51(2), pages 328-356.
    12. Sun, Ang & Zhao, Yaohui, 2016. "Divorce, abortion, and the child sex ratio: The impact of divorce reform in China," Journal of Development Economics, Elsevier, vol. 120(C), pages 53-69.
    13. Mellace, Giovanni & Ventura, Marco, 2019. "Intended and unintended effects of public incentives for innovation. Quasi-experimental evidence from Italy," Discussion Papers on Economics 9/2019, University of Southern Denmark, Department of Economics.
    14. Adam C. Sales & Ben B. Hansen, 2020. "Limitless Regression Discontinuity," Journal of Educational and Behavioral Statistics, , vol. 45(2), pages 143-174, April.
    15. Eduardo Fé & Bruce Hollingsworth, 2016. "Short- and long-run estimates of the local effects of retirement on health," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 179(4), pages 1051-1067, October.
    16. Quaresima Federico & Santolini Raffaella & Fiorillo Fabio, 2020. "Political affiliation in post-parliamentary careers in Italian public enterprises," German Economic Review, De Gruyter, vol. 21(1), pages 35-64, April.
    17. Berger, Melissa & Fellner-Röhling, Gerlinde & Sausgruber, Rupert & Traxler, Christian, 2016. "Higher taxes, more evasion? Evidence from border differentials in TV license fees," Journal of Public Economics, Elsevier, vol. 135(C), pages 74-86.
    18. Otávio Bartalotti, 2013. "Theory and Practice of Inference in Regression Discontinuity: A Fixed-Bandwidth Asymptotics Approach," Working Papers 1302, Tulane University, Department of Economics, revised Nov 2013.
    19. Granato, Silvia & Havari, Enkelejda & Mazzarella, Gianluca & Schnepf, Sylke V., 2024. "Study abroad programmes and student outcomes: Evidence from Erasmus," Economics of Education Review, Elsevier, vol. 99(C).
    20. Teevrat Garg & Maulik Jagnani & Hemant K. Pullabhotla, 2022. "Structural transformation and environmental externalities," Papers 2212.02664, arXiv.org.
    21. repec:osf:socarx:2qvwb_v1 is not listed on IDEAS
    22. Felix Thoemmes & Wang Liao & Ze Jin, 2017. "The Analysis of the Regression-Discontinuity Design in R," Journal of Educational and Behavioral Statistics, , vol. 42(3), pages 341-360, June.

    More about this item

    Keywords

    ;
    ;
    ;
    ;
    ;

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:mpr:mprres:f0ff8f86e3c34dc8baaf22b564ed8081. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Joanne Pfleiderer or Cindy George The email address of this maintainer does not seem to be valid anymore. Please ask Cindy George to update the entry or send us the correct address (email available below). General contact details of provider: https://edirc.repec.org/data/mathius.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.