IDEAS home Printed from https://ideas.repec.org/a/aea/aejapp/v11y2019i3p382-423.html
   My bibliography  Save this article

The Causes and Consequences of Test Score Manipulation: Evidence from the New York Regents Examinations

Author

Listed:
  • Thomas S. Dee
  • Will Dobbie
  • Brian A. Jacob
  • Jonah Rockoff

Abstract

We show that the design and decentralized scoring of New York's high school exit exams—the Regents Examinations—led to systematic manipulation of test scores just below important proficiency cutoffs. Exploiting a series of reforms that eliminated score manipulation, we find heterogeneous effects of test score manipulation on academic outcomes. While inflating a score increases the probability of a student graduating from high school by about 17 percentage points, the probability of taking advanced coursework declines by roughly 10 percentage points. We argue that these results are consistent with test score manipulation helping less advanced students on the margin of dropping out but hurting more advanced students that are not pushed to gain a solid foundation in the introductory material.

Suggested Citation

  • Thomas S. Dee & Will Dobbie & Brian A. Jacob & Jonah Rockoff, 2019. "The Causes and Consequences of Test Score Manipulation: Evidence from the New York Regents Examinations," American Economic Journal: Applied Economics, American Economic Association, vol. 11(3), pages 382-423, July.
    • Handle: RePEc:aea:aejapp:v:11:y:2019:i:3:p:382-423
    Note: DOI: 10.1257/app.20170520
    as

    Download full text from publisher

    File URL: https://www.aeaweb.org/doi/10.1257/app.20170520
    Download Restriction: no
    File URL: https://www.aeaweb.org/doi/10.1257/app.20170520.data
    Download Restriction: no
    File URL: https://www.aeaweb.org/doi/10.1257/app.20170520.appx
    Download Restriction: no
    File URL: https://www.aeaweb.org/doi/10.1257/app.20170520.ds
    Download Restriction: Access to full text is restricted to AEA members and institutional subscribers.
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Joshua D. Angrist & Alan B. Keueger, 1991. "Does Compulsory School Attendance Affect Schooling and Earnings?," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 106(4), pages 979-1014.
    2. Jörn-Steffen Pischke & Till von Wachter, 2008. "Zero Returns to Compulsory Schooling in Germany: Evidence and Interpretation," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 592-598, August.
    3. Jaeger, David A & Page, Marianne E, 1996. "Degrees Matter: New Evidence on Sheepskin Effects in the Returns to Education," The Review of Economics and Statistics, MIT Press, vol. 78(4), pages 733-740, November.
    4. Ou, Dongshu, 2010. "To leave or not to leave? A regression discontinuity analysis of the impact of failing the high school exit exam," Economics of Education Review, Elsevier, vol. 29(2), pages 171-186, April.
    5. Victor Lavy, 2009. "Performance Pay and Teachers' Effort, Productivity, and Grading Ethics," American Economic Review, American Economic Association, vol. 99(5), pages 1979-2011, December.
    6. Derek Neal, 2013. "The Consequences of Using one Assessment System to Pursue two Objectives," The Journal of Economic Education, Taylor & Francis Journals, vol. 44(4), pages 339-352, October.
    7. Giorgio Brunello & Margherita Fort & Guglielmo Weber, 2009. "Changes in Compulsory Schooling, Education and the Distribution of Wages in Europe," Economic Journal, Royal Economic Society, vol. 119(536), pages 516-539, March.
    8. Raj Chetty & John N. Friedman & Tore Olsen & Luigi Pistaferri, 2011. "Adjustment Costs, Firm Responses, and Micro vs. Macro Labor Supply Elasticities: Evidence from Danish Tax Records," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(2), pages 749-804.
    9. Hinnerich, Björn Tyrefors & Höglin, Erik & Johannesson, Magnus, 2011. "Are boys discriminated in Swedish high schools?," Economics of Education Review, Elsevier, vol. 30(4), pages 682-690, August.
    10. Julie Berry Cullen & Randall Reback, 2006. "Tinkering Toward Accolades: School Gaming Under a Performance Accountability System," NBER Working Papers 12286, National Bureau of Economic Research, Inc.
    11. Brian A. Jacob & Steven D. Levitt, 2003. "Rotten Apples: An Investigation of the Prevalence and Predictors of Teacher Cheating," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 118(3), pages 843-877.
    12. Lavy, Victor & Sand, Edith, 2015. "On The Origins of Gender Human Capital Gaps: Short and Long Term Consequences of Teachers’ Stereotypical Biases," Economic Research Papers 270009, University of Warwick - Department of Economics.
    13. Derek Neal, 2013. "The Consequences of Using One Assessment System To Pursue Two Objectives," NBER Working Papers 19214, National Bureau of Economic Research, Inc.
    14. Lavy, Victor, 2008. "Do gender stereotypes reduce girls' or boys' human capital outcomes? Evidence from a natural experiment," Journal of Public Economics, Elsevier, vol. 92(10-11), pages 2083-2105, October.
    15. Roland G. Fryer, 2013. "Teacher Incentives and Student Achievement: Evidence from New York City Public Schools," Journal of Labor Economics, University of Chicago Press, vol. 31(2), pages 373-407.
    16. Papay, John P. & Willett, John B. & Murnane, Richard J., 2011. "Extending the regression-discontinuity approach to multiple assignment variables," Journal of Econometrics, Elsevier, vol. 161(2), pages 203-207, April.
    17. Petra Persson, 2020. "Social Insurance and the Marriage Market," Journal of Political Economy, University of Chicago Press, vol. 128(1), pages 252-300.
    18. Emmanuel Saez, 2010. "Do Taxpayers Bunch at Kink Points?," American Economic Journal: Economic Policy, American Economic Association, vol. 2(3), pages 180-212, August.
    19. Derek Neal & Diane Whitmore Schanzenbach, 2010. "Left Behind by Design: Proficiency Counts and Test-Based Accountability," The Review of Economics and Statistics, MIT Press, vol. 92(2), pages 263-283, May.
    20. Henrik J. Kleven & Mazhar Waseem, 2013. "Using Notches to Uncover Optimization Frictions and Structural Elasticities: Theory and Evidence from Pakistan," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 128(2), pages 669-723.
    21. Oreopoulos, Philip, 2007. "Do dropouts drop out too soon? Wealth, health and happiness from compulsory schooling," Journal of Public Economics, Elsevier, vol. 91(11-12), pages 2213-2229, December.
    22. Simon Burgess & Ellen Greaves, 2013. "Test Scores, Subjective Assessment, and Stereotyping of Ethnic Minorities," Journal of Labor Economics, University of Chicago Press, vol. 31(3), pages 535-576.
    23. Damon Clark & Paco Martorell, 2014. "The Signaling Value of a High School Diploma," Journal of Political Economy, University of Chicago Press, vol. 122(2), pages 282-318.
    24. David N. Figlio & Lawrence S. Getzler, 2002. "Accountability , Ability and Disability: Gaming the System," NBER Working Papers 9307, National Bureau of Economic Research, Inc.
    25. Serena Canaan & Pierre Mouganie, 2018. "Returns to Education Quality for Low-Skilled Students: Evidence from a Discontinuity," Journal of Labor Economics, University of Chicago Press, vol. 36(2), pages 395-436.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Figlio, D. & Karbownik, K. & Salvanes, K.G., 2016. "Education Research and Administrative Data," Handbook of the Economics of Education,, Elsevier.
    2. Rebecca Diamond & Petra Persson, 2016. "The Long-term Consequences of Teacher Discretion in Grading of High-stakes Tests," NBER Working Papers 22207, National Bureau of Economic Research, Inc.
    3. Cuesta, José Ignacio & González, Felipe & Larroulet Philippi, Cristian, 2020. "Distorted quality signals in school markets," Journal of Development Economics, Elsevier, vol. 147(C).
    4. Terrier, Camille, 2020. "Boys lag behind: How teachers’ gender biases affect student achievement," Economics of Education Review, Elsevier, vol. 77(C).
    5. Gershenson, Seth & Holt, Stephen B. & Papageorge, Nicholas W., 2015. "Who Believes in Me? The Effect of Student-Teacher Demographic Match on Teacher Expectations," IZA Discussion Papers 9202, Institute of Labor Economics (IZA).
    6. Brehm, Margaret & Imberman, Scott A. & Lovenheim, Michael F., 2017. "Achievement effects of individual performance incentives in a teacher merit pay tournament," Labour Economics, Elsevier, vol. 44(C), pages 133-150.
    7. Meng, Xin & Zhao, Guochang, 2021. "The long shadow of a large scale education interruption: The intergenerational effect," Labour Economics, Elsevier, vol. 71(C).
    8. Huong Thu Le & Ha Trong Nguyen, 2018. "The evolution of the gender test score gap through seventh grade: new insights from Australia using unconditional quantile regression and decomposition," IZA Journal of Labor Economics, Springer;Forschungsinstitut zur Zukunft der Arbeit GmbH (IZA), vol. 7(1), pages 1-42, December.
    9. Murat G. Kırdar & Meltem Dayıoğlu & İsmet Koç, 2016. "Does Longer Compulsory Education Equalize Schooling by Gender and Rural/Urban Residence?," The World Bank Economic Review, World Bank, vol. 30(3), pages 549-579.
    10. Joshua D. Angrist & Erich Battistin & Daniela Vuri, 2017. "In a Small Moment: Class Size and Moral Hazard in the Italian Mezzogiorno," American Economic Journal: Applied Economics, American Economic Association, vol. 9(4), pages 216-249, October.
    11. Derek Neal, 2011. "The Design of Performance Pay in Education," NBER Working Papers 16710, National Bureau of Economic Research, Inc.
    12. Barrera-Osorio, Felipe & Cilliers, Jacobus & Cloutier, Marie-Hélène & Filmer, Deon, 2022. "Heterogenous teacher effects of two incentive schemes: Evidence from a low-income country," Journal of Development Economics, Elsevier, vol. 156(C).
    13. Isaac Mbiti & Mauricio Romero & Youdi Schipper, 2023. "Designing Effective Teacher Performance Pay Programs: Experimental Evidence from Tanzania," The Economic Journal, Royal Economic Society, vol. 133(653), pages 1968-2000.
    14. Cornelisz, Ilja & Meeter, Martijn & van Klaveren, Chris, 2019. "Educational equity and teacher discretion effects in high stake exams," Economics of Education Review, Elsevier, vol. 73(C).
    15. Fan, Elliott & Lien, Hsienming & Ma, Ching-to Albert, 2019. "Uterus at a price: Disability insurance and hysterectomy," Journal of Health Economics, Elsevier, vol. 66(C), pages 1-17.
    16. Ulrik Hvidman & Hans Henrik Sievertsen, 2021. "High-Stakes Grades and Student Behavior," Journal of Human Resources, University of Wisconsin Press, vol. 56(3), pages 821-849.
    17. Bahadır Dursun & Resul Cesur, 2016. "Transforming lives: the impact of compulsory schooling on hope and happiness," Journal of Population Economics, Springer;European Society for Population Economics, vol. 29(3), pages 911-956, July.
    18. Camille Terrier, 2014. "Giving a Little Help to Girls? Evidence on Grade Discrimination and its Effect on Students Achievement," Working Papers hal-01080834, HAL.
    19. de New, Sonja C. & Schurer, Stefanie & Sulzmaier, Dominique, 2021. "Gender differences in the lifecycle benefits of compulsory schooling policies," European Economic Review, Elsevier, vol. 140(C).
    20. Rapoport, Benoît & Thibout, Claire, 2018. "Why do boys and girls make different educational choices? The influence of expected earnings and test scores," Economics of Education Review, Elsevier, vol. 62(C), pages 205-229.

    More about this item

    JEL classification:

    • H75 - Public Economics - - State and Local Government; Intergovernmental Relations - - - State and Local Government: Health, Education, and Welfare
    • I21 - Health, Education, and Welfare - - Education - - - Analysis of Education
    • I28 - Health, Education, and Welfare - - Education - - - Government Policy

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:aea:aejapp:v:11:y:2019:i:3:p:382-423. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Michael P. Albert (email available below). General contact details of provider: https://edirc.repec.org/data/aeaaaea.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.