IDEAS home Printed from https://ideas.repec.org/a/oup/qjecon/v132y2017i2p871-919..html
   My bibliography  Save this article

Leveraging Lotteries for School Value-Added: Testing and Estimation

Author

Listed:
  • Joshua D. Angrist
  • Peter D. Hull
  • Parag A. Pathak
  • Christopher R. Walters

Abstract

Conventional value-added models (VAMs) compare average test scores across schools after regression-adjusting for students’ demographic characteristics and previous scores. This article tests for VAM bias using a procedure that asks whether VAM estimates accurately predict the achievement consequences of random assignment to specific schools. Test results from admissions lotteries in Boston suggest conventional VAM estimates are biased, a finding that motivates the development of a hierarchical model describing the joint distribution of school value-added, bias, and lottery compliance. We use this model to assess the substantive importance of bias in conventional VAM estimates and to construct hybrid value-added estimates that optimally combine ordinary least squares and lottery-based estimates of VAM parameters. The hybrid estimation strategy provides a general recipe for combining nonexperimental and quasi-experimental estimates. While still biased, hybrid school value-added estimates have lower mean squared error than conventional VAM estimates. Simulations calibrated to the Boston data show that, bias notwithstanding, policy decisions based on conventional VAMs that control for lagged achievement are likely to generate substantial achievement gains. Hybrid estimates that incorporate lotteries yield further gains.

Suggested Citation

  • Joshua D. Angrist & Peter D. Hull & Parag A. Pathak & Christopher R. Walters, 2017. "Leveraging Lotteries for School Value-Added: Testing and Estimation," The Quarterly Journal of Economics, Oxford University Press, vol. 132(2), pages 871-919.
  • Handle: RePEc:oup:qjecon:v:132:y:2017:i:2:p:871-919.
    as

    Download full text from publisher

    File URL: http://hdl.handle.net/10.1093/qje/qjx001
    Download Restriction: Access to full text is restricted to subscribers.

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Atila Abdulkadiroğlu & Joshua D. Angrist & Peter D. Hull & Parag A. Pathak, 2016. "Charters without Lotteries: Testing Takeovers in New Orleans and Boston," American Economic Review, American Economic Association, vol. 106(7), pages 1878-1920, July.
    2. White, Halbert, 1980. "A Heteroskedasticity-Consistent Covariance Matrix Estimator and a Direct Test for Heteroskedasticity," Econometrica, Econometric Society, vol. 48(4), pages 817-838, May.
    3. Raj Chetty & John N. Friedman & Nathaniel Hilger & Emmanuel Saez & Diane Whitmore Schanzenbach & Danny Yagan, 2011. "How Does Your Kindergarten Classroom Affect Your Earnings? Evidence from Project Star," The Quarterly Journal of Economics, Oxford University Press, vol. 126(4), pages 1593-1660.
    4. Judge G.G. & Mittelhammer R.C., 2004. "A Semiparametric Basis for Combining Estimation Problems Under Quadratic Loss," Journal of the American Statistical Association, American Statistical Association, vol. 99, pages 479-487, January.
    5. McFadden, Daniel, 1989. "A Method of Simulated Moments for Estimation of Discrete Response Models without Numerical Integration," Econometrica, Econometric Society, vol. 57(5), pages 995-1026, September.
    6. David J. Deming, 2014. "Using School Choice Lotteries to Test Measures of School Effectiveness," American Economic Review, American Economic Association, vol. 104(5), pages 406-411, May.
    7. Cory Koedel & Julian R. Betts, 2011. "Does Student Sorting Invalidate Value-Added Models of Teacher Effectiveness? An Extended Analysis of the Rothstein Critique," Education Finance and Policy, MIT Press, vol. 6(1), pages 18-42, January.
    8. Atila Abdulkadiroğlu & Joshua D. Angrist & Susan M. Dynarski & Thomas J. Kane & Parag A. Pathak, 2011. "Accountability and Flexibility in Public Schools: Evidence from Boston's Charters And Pilots," The Quarterly Journal of Economics, Oxford University Press, vol. 126(2), pages 699-748.
    9. Josh Kinsler, 2012. "Assessing Rothstein's critique of teacher value‐added models," Quantitative Economics, Econometric Society, vol. 3(2), pages 333-362, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. repec:aea:aecrev:v:109:y:2019:i:4:p:1426-60 is not listed on IDEAS
    2. Hinnerich, Björn Tyrefors & Vlachos, Jonas, 2017. "The impact of upper-secondary voucher school attendance on student achievement. Swedish evidence using external and internal evaluations," Labour Economics, Elsevier, vol. 47(C), pages 1-14.
    3. Adam Kapor & Christopher A. Neilson & Seth D. Zimmerman, 2017. "Heterogeneous Beliefs and School Choice Mechanisms," Working Papers 612, Princeton University, Department of Economics, Industrial Relations Section..
    4. Brandt Kasper, 2018. "Private beats public: A flexible value-added model with Tanzanian school switchers," WIDER Working Paper Series 81, World Institute for Development Economic Research (UNU-WIDER).
    5. Schiltz, Fritz & Sestito, Paolo & Agasisti, Tommaso & De Witte, Kristof, 2018. "The added value of more accurate predictions for school rankings," Economics of Education Review, Elsevier, vol. 67(C), pages 207-215.
    6. Yusuki Narita, 2016. "(Non)Randomization: A Theory of Quasi-Experimental Evaluation of School Quality," Cowles Foundation Discussion Papers 2056R, Cowles Foundation for Research in Economics, Yale University, revised Aug 2017.
    7. Mira Frick & Ryota Iijima & Tomasz Strzalecki, 2017. "Dynamic Random Utility," Cowles Foundation Discussion Papers 2092, Cowles Foundation for Research in Economics, Yale University.
    8. Karthik Muralidharan & Abhijeet Singh & Alejandro J. Ganimian, 2019. "Disrupting Education? Experimental Evidence on Technology-Aided Instruction in India," American Economic Review, American Economic Association, vol. 109(4), pages 1426-1460, April.
    9. Jean-William Laliberté, "undated". "Long-term Contextual Effects in Education: Schools and Neighborhoods," Working Papers 2019-01, Department of Economics, University of Calgary.
    10. Jesse Rothstein & Till von Wachter, 2016. "Social Experiments in the Labor Market," NBER Working Papers 22585, National Bureau of Economic Research, Inc.
    11. Doris, Aedin & O'Neill, Donal & Sweetman, Olive, 2019. "Good Schools or Good Students? The Importance of Selectivity for School Rankings," IZA Discussion Papers 12459, Institute of Labor Economics (IZA).
    12. Joshua Angrist & Peter Hull & Parag Pathak & Christopher Walters, 2016. "Interpreting Tests of School VAM Validity," American Economic Review, American Economic Association, vol. 106(5), pages 388-392, May.
    13. repec:wly:emetrp:v:85:y:2017:i::p:1373-1432 is not listed on IDEAS
    14. Stéphane Bonhomme & Martin Weidner, 2018. "Minimizing sensitivity to model misspecification," CeMMAP working papers CWP59/18, Centre for Microdata Methods and Practice, Institute for Fiscal Studies.
    15. Nirav Mehta, 2017. "Measuring Quality for Use in Incentive Schemes: The Case of "Shrinkage" Estimators," University of Western Ontario, Centre for Human Capital and Productivity (CHCP) Working Papers 201725, University of Western Ontario, Centre for Human Capital and Productivity (CHCP).
    16. Raj Chetty & John N. Friedman & Jonah Rockoff, 2016. "Using Lagged Outcomes to Evaluate Bias in Value-Added Models," American Economic Review, American Economic Association, vol. 106(5), pages 393-399, May.
    17. Atila Abdulkadiroğlu & Joshua D. Angrist & Yusuke Narita & Parag A. Pathak, 2017. "Research Design Meets Market Design: Using Centralized Assignment for Impact Evaluation," Econometrica, Econometric Society, vol. 85, pages 1373-1432, September.
    18. Clement de Chaisemartin & Luc Behaghel, 2015. "Estimating the effect of treatments allocated by randomized waiting lists," Papers 1511.01453, arXiv.org, revised Oct 2018.
    19. Amy Finkelstein & Matthew Gentzkow & Heidi L. Williams, 2019. "Place-Based Drivers of Mortality: Evidence from Migration," NBER Working Papers 25975, National Bureau of Economic Research, Inc.
    20. Yusuki Narita, 2016. "(Non)Randomization: A Theory of Quasi-Experimental Evaluation of School Quality," Cowles Foundation Discussion Papers 2056, Cowles Foundation for Research in Economics, Yale University.
    21. repec:eee:injoed:v:60:y:2018:i:c:p:33-50 is not listed on IDEAS
    22. Atila Abdulkadiroğlu & Parag A. Pathak & Christopher R. Walters, 2018. "Free to Choose: Can School Choice Reduce Student Achievement?," American Economic Journal: Applied Economics, American Economic Association, vol. 10(1), pages 175-206, January.
    23. repec:tpr:edfpol:v:12:y:2017:i:4:p:468-491 is not listed on IDEAS

    More about this item

    JEL classification:

    • I20 - Health, Education, and Welfare - - Education - - - General
    • J24 - Labor and Demographic Economics - - Demand and Supply of Labor - - - Human Capital; Skills; Occupational Choice; Labor Productivity
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:oup:qjecon:v:132:y:2017:i:2:p:871-919.. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Oxford University Press) The email address of this maintainer does not seem to be valid anymore. Please ask Oxford University Press to update the entry or send us the correct email address or (Christopher F. Baum). General contact details of provider: .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.