IDEAS home Printed from https://ideas.repec.org/p/hka/wpaper/2020-085.html
   My bibliography  Save this paper

A Theory of Quasi-Experimental Evaluation of School Quality

Author

Listed:
  • Yusuke Narita

    (Massachusetts Institute of Technology)

Abstract

Many centralized school admissions systems use lotteries to ration limited seats at oversubscribed schools. The resulting random assignment is used by empirical researchers to identify the effects of schools on outcomes like test scores. I first find that the two most popular empirical research designs may not successfully extract a random assignment of applicants to schools. When do the research designs overcome this problem? I show the following main results for a class of data-generating mechanisms containing those used in practice: The “first-choice†research design extracts a random assignment under a mechanism if the mechanism is strategy-proof for schools. In contrast, the other “qualification instrument†research design does not necessarily extract a random assignment under any mechanism. The former research design is therefore more compelling than the latter. Many applications of the two research designs need some implicit assumption, such as large-sample approximately random assignment, to justify their empirical strategy.

Suggested Citation

  • Yusuke Narita, 2020. "A Theory of Quasi-Experimental Evaluation of School Quality," Working Papers 2020-085, Human Capital and Economic Opportunity Working Group.
    • Handle: RePEc:hka:wpaper:2020-085
    Note: MIP
    as

    Download full text from publisher

    File URL: http://humcap.uchicago.edu/RePEc/hka/wpaper/Narita_2020_theory-quasi-experimental-school-quality.pdf
    File Function: First version, June 14, 2020
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Justine S. Hastings & Christopher A. Neilson & Seth D. Zimmerman, 2013. "Are Some Degrees Worth More than Others? Evidence from college admission cutoffs in Chile," NBER Working Papers 19241, National Bureau of Economic Research, Inc.
    2. Jackson, C. Kirabo, 2012. "Single-sex schools, student achievement, and course selection: Evidence from rule-based student assignments in Trinidad and Tobago," Journal of Public Economics, Elsevier, vol. 96(1), pages 173-187.
    3. Adrienne M. Lucas & Isaac M. Mbiti, 2014. "Effects of School Quality on Student Achievement: Discontinuity Evidence from Kenya," American Economic Journal: Applied Economics, American Economic Association, vol. 6(3), pages 234-263, July.
    4. David J. Deming, 2011. "Better Schools, Less Crime?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(4), pages 2063-2115.
    5. Atı̇la Abdulkadı̇roğlu & Joshua D. Angrist & Yusuke Narita & Parag Pathak, 2022. "Breaking Ties: Regression Discontinuity Design Meets Market Design," Econometrica, Econometric Society, vol. 90(1), pages 117-151, January.
    6. Atila Abdulkadroǧlu & Joshua D. Angrist & Yusuke Narita & Parag A. Pathak & Roman A. Zarate, 2017. "Regression Discontinuity in Serial Dictatorship: Achievement Effects at Chicago's Exam Schools," American Economic Review, American Economic Association, vol. 107(5), pages 240-245, May.
    7. Fort, Margherita & Zanella, Giulio, 2019. "Cognitive and non-cognitive costs of daycare 0–2 for children in advantaged families," CEPR Discussion Papers 11120, C.E.P.R. Discussion Papers.
    8. Caterina Calsamiglia & Chao Fu & Maia Güell, 2014. "Structural Estimation of a Model of School Choices: the Boston Mechanism vs. Its Alternatives," Working Papers 811, Barcelona School of Economics.
    9. Joshua D. Angrist & Jörn-Steffen Pischke, 2009. "Mostly Harmless Econometrics: An Empiricist's Companion," Economics Books, Princeton University Press, edition 1, number 8769.
    10. , A. & ,, 2011. "Lotteries in student assignment: An equivalence result," Theoretical Economics, Econometric Society, vol. 6(1), January.
    11. Justine S. Hastings & Christopher A. Neilson & Seth D. Zimmerman, 2012. "The Effect of School Choice on Intrinsic Motivation and Academic Outcomes," Working Papers 2012-3, Princeton University. Economics Department..
    12. David J. Deming & Justine S. Hastings & Thomas J. Kane & Douglas O. Staiger, 2014. "School Choice, School Quality, and Postsecondary Attainment," American Economic Review, American Economic Association, vol. 104(3), pages 991-1013, March.
    13. C. Kirabo Jackson, 2010. "Do Students Benefit from Attending Better Schools? Evidence from Rule-based Student Assignments in Trinidad and Tobago," Economic Journal, Royal Economic Society, vol. 120(549), pages 1399-1429, December.
    14. Fuhito Kojima & Parag A. Pathak, 2009. "Incentives and Stability in Large Two-Sided Matching Markets," American Economic Review, American Economic Association, vol. 99(3), pages 608-627, June.
    15. Alexandre Belloni & Victor Chernozhukov & Christian Hansen, 2014. "High-Dimensional Methods and Inference on Structural and Treatment Effects," Journal of Economic Perspectives, American Economic Association, vol. 28(2), pages 29-50, Spring.
    16. Yusuke Narita, 2018. "Experiment-as-Market: Incorporating Welfare into Randomized Controlled Trials," Cowles Foundation Discussion Papers 2127r, Cowles Foundation for Research in Economics, Yale University, revised May 2019.
    17. Gary Chamberlain, 1980. "Analysis of Covariance with Qualitative Data," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 47(1), pages 225-238.
    18. Adam J. Kapor & Christopher A. Neilson & Seth D. Zimmerman, 2020. "Heterogeneous Beliefs and School Choice Mechanisms," American Economic Review, American Economic Association, vol. 110(5), pages 1274-1315, May.
    19. Roth, Alvin E & Sotomayor, Marilda, 1989. "The College Admissions Problem Revisited," Econometrica, Econometric Society, vol. 57(3), pages 559-570, May.
    20. Ergin, Haluk & Sonmez, Tayfun, 2006. "Games of school choice under the Boston mechanism," Journal of Public Economics, Elsevier, vol. 90(1-2), pages 215-237, January.
    21. Howard S. Bloom & Rebecca Unterman, 2014. "Can Small High Schools of Choice Improve Educational Prospects for Disadvantaged Students?," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 33(2), pages 290-319, March.
    22. Shapley, Lloyd & Scarf, Herbert, 1974. "On cores and indivisibility," Journal of Mathematical Economics, Elsevier, vol. 1(1), pages 23-37, March.
    23. Ivan A. Canay & Joseph P. Romano & Azeem M. Shaikh, 2017. "Randomization Tests Under an Approximate Symmetry Assumption," Econometrica, Econometric Society, vol. 85, pages 1013-1030, May.
    24. He, Yinghua, 2012. "Gaming the Boston School Choice Mechanism in Beijing," TSE Working Papers 12-345, Toulouse School of Economics (TSE).
    25. Alexander Shapiro & Jos Berge, 2002. "Statistical inference of minimum rank factor analysis," Psychometrika, Springer;The Psychometric Society, vol. 67(1), pages 79-94, March.
    26. Atila Abdulkadiroglu & Tayfun Sönmez, 2003. "School Choice: A Mechanism Design Approach," American Economic Review, American Economic Association, vol. 93(3), pages 729-747, June.
    27. repec:oup:restud:v:84:y::i:1:p:444-463. is not listed on IDEAS
    28. Hatfield, John William & Kojima, Fuhito & Narita, Yusuke, 2016. "Improving schools through school choice: A market design approach," Journal of Economic Theory, Elsevier, vol. 166(C), pages 186-211.
    29. Cristian Pop-Eleches & Miguel Urquiola, 2013. "Going to a Better School: Effects and Behavioral Responses," American Economic Review, American Economic Association, vol. 103(4), pages 1289-1324, June.
    30. Nikhil Agarwal & Paulo Somaini, 2018. "Demand Analysis Using Strategic Reports: An Application to a School Choice Mechanism," Econometrica, Econometric Society, vol. 86(2), pages 391-444, March.
    31. Azevedo, Eduardo M., 2014. "Imperfect competition in two-sided matching markets," Games and Economic Behavior, Elsevier, vol. 83(C), pages 207-223.
    32. Eduardo M. Azevedo & Jacob D. Leshno, 2016. "A Supply and Demand Framework for Two-Sided Matching Markets," Journal of Political Economy, University of Chicago Press, vol. 124(5), pages 1235-1268.
    33. Balinski, Michel & Sonmez, Tayfun, 1999. "A Tale of Two Mechanisms: Student Placement," Journal of Economic Theory, Elsevier, vol. 84(1), pages 73-94, January.
    34. Elliott Peranson & Alvin E. Roth, 1999. "The Redesign of the Matching Market for American Physicians: Some Engineering Aspects of Economic Design," American Economic Review, American Economic Association, vol. 89(4), pages 748-780, September.
    35. Yan Chen & Onur Kesten, 2017. "Chinese College Admissions and School Choice Reforms: A Theoretical Analysis," Journal of Political Economy, University of Chicago Press, vol. 125(1), pages 99-139.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Yusuke Narita & Kohei Yata, 2021. "Algorithm is Experiment: Machine Learning, Market Design, and Policy Eligibility Rules," Working Papers 2021-022, Human Capital and Economic Opportunity Working Group.
    2. Atı̇la Abdulkadı̇roğlu & Joshua D. Angrist & Yusuke Narita & Parag Pathak, 2022. "Breaking Ties: Regression Discontinuity Design Meets Market Design," Econometrica, Econometric Society, vol. 90(1), pages 117-151, January.
    3. Narita, Yusuke & Yata, Kohei, 2022. "Algorithm is Experiment: Machine Learning, Market Design, and Policy Eligibility Rules," CEI Working Paper Series 2021-05, Center for Economic Institutions, Institute of Economic Research, Hitotsubashi University.
    4. Narita, Yusuke & Yata, Kohei, 2022. "Algorithm is Experiment: Machine Learning, Market Design, and Policy Eligibility Rules," Discussion Paper Series 730, Institute of Economic Research, Hitotsubashi University.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Yusuke Narita, 2021. "A Theory of Quasi-Experimental Evaluation of School Quality," Management Science, INFORMS, vol. 67(8), pages 4982-5010, August.
    2. Yusuki Narita, 2016. "(Non)Randomization: A Theory of Quasi-Experimental Evaluation of School Quality," Cowles Foundation Discussion Papers 2056R, Cowles Foundation for Research in Economics, Yale University, revised Aug 2017.
    3. Atı̇la Abdulkadı̇roğlu & Joshua D. Angrist & Yusuke Narita & Parag Pathak, 2022. "Breaking Ties: Regression Discontinuity Design Meets Market Design," Econometrica, Econometric Society, vol. 90(1), pages 117-151, January.
    4. Atila Abdulkadiroğlu & Joshua D. Angrist & Yusuke Narita & Parag A. Pathak, 2017. "Research Design Meets Market Design: Using Centralized Assignment for Impact Evaluation," Econometrica, Econometric Society, vol. 85, pages 1373-1432, September.
    5. Gabrielle Fack & Julien Grenet & Yinghua He, 2019. "Beyond Truth-Telling: Preference Estimation with Centralized School Choice and College Admissions," American Economic Review, American Economic Association, vol. 109(4), pages 1486-1529, April.
    6. Alvin Roth, 2008. "Deferred acceptance algorithms: history, theory, practice, and open questions," International Journal of Game Theory, Springer;Game Theory Society, vol. 36(3), pages 537-569, March.
    7. John William Hatfield & Fuhito Kojima & Yusuke Narita, 2011. "Promoting School Competition Through School Choice: A Market Design Approach," Working Papers 2011-018, Human Capital and Economic Opportunity Working Group.
    8. Abdulkadiroglu, Atila & Andersson, Tommy, 2022. "School Choice," Working Papers 2022:4, Lund University, Department of Economics.
    9. Cao, Yuan, 2020. "Centralized assignment mechanisms and assortative matching: Evidence from Chinese universities," Journal of Economic Behavior & Organization, Elsevier, vol. 177(C), pages 255-276.
    10. Li, Mengling, 2020. "Ties matter: Improving efficiency in course allocation by allowing ties," Journal of Economic Behavior & Organization, Elsevier, vol. 178(C), pages 354-384.
    11. Min Zhu, 2013. "College Admissions in China : A Mechanism Design Perspective," Working Papers 1327, Groupe d'Analyse et de Théorie Economique Lyon St-Étienne (GATE Lyon St-Étienne), Université de Lyon.
    12. Zhu, Min, 2014. "College admissions in China: A mechanism design perspective," China Economic Review, Elsevier, vol. 30(C), pages 618-631.
    13. Ha, Wei & Kang, Le & Song, Yang, 2020. "College matching mechanisms and matching stability: Evidence from a natural experiment in China," Journal of Economic Behavior & Organization, Elsevier, vol. 175(C), pages 206-226.
    14. Thierry Magnac, 2018. "Quels étudiants pour quelles universités ? Analyses empiriques de mécanismes d’allocation centralisée," Revue économique, Presses de Sciences-Po, vol. 69(5), pages 683-708.
    15. Hatfield, John William & Kojima, Fuhito & Narita, Yusuke, 2016. "Improving schools through school choice: A market design approach," Journal of Economic Theory, Elsevier, vol. 166(C), pages 186-211.
    16. Nikhil Agarwal & Paulo Somaini, 2018. "Demand Analysis Using Strategic Reports: An Application to a School Choice Mechanism," Econometrica, Econometric Society, vol. 86(2), pages 391-444, March.
    17. Kamada, Yuichiro & Kojima, Fuhito, 2017. "Stability concepts in matching under distributional constraints," Journal of Economic Theory, Elsevier, vol. 168(C), pages 107-142.
    18. Atila Abdulkadiroglu & Joshua D. Angrist & Yusuke Narita & Parag A. Pathak, 2017. "Impact Evaluation in Matching Markets with General Tie-Breaking," NBER Working Papers 24172, National Bureau of Economic Research, Inc.
    19. Min Zhu, 2013. "College Admissions in China : A Mechanism Design Perspective," Working Papers halshs-00860931, HAL.
    20. Eduardo M. Azevedo & Jacob D. Leshno, 2016. "A Supply and Demand Framework for Two-Sided Matching Markets," Journal of Political Economy, University of Chicago Press, vol. 124(5), pages 1235-1268.

    More about this item

    Keywords

    market design; natural experiment; school effectiveness;
    All these keywords.

    JEL classification:

    • C93 - Mathematical and Quantitative Methods - - Design of Experiments - - - Field Experiments
    • D47 - Microeconomics - - Market Structure, Pricing, and Design - - - Market Design
    • I24 - Health, Education, and Welfare - - Education - - - Education and Inequality

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:hka:wpaper:2020-085. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Jennifer Pachon (email available below). General contact details of provider: https://edirc.repec.org/data/mfichus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.