IDEAS home Printed from https://ideas.repec.org/p/iza/izadps/dp15321.html
   My bibliography  Save this paper

What Can We Learn from Student Performance Measures? Identifying Treatment in the Presence of Curves and Letter Grades

Author

Listed:
  • Waddell, Glen R.

    (University of Oregon)

  • Putz, Jenni

    (University of Oregon)

Abstract

Grade-based performance measures are often relied on when considering the efficacy of education-related policy interventions. Yet, it is common for measures of student performance to be subjected to curves and discretized through letter-grade transformations. We show how transformed grades systematically challenge causal identification. Even without explicit curving, transformations to letter grade are particularly problematic and yield treatment estimates that are weighted combinations of inflated responsiveness around letter thresholds and "zeros" away from these thresholds. Curving practices can also introduce false patterns of treatment heterogeneity, attenuating measured responses to treatment among high-performing students, for example, or inflating measured responses among low-performing students.

Suggested Citation

  • Waddell, Glen R. & Putz, Jenni, 2022. "What Can We Learn from Student Performance Measures? Identifying Treatment in the Presence of Curves and Letter Grades," IZA Discussion Papers 15321, Institute of Labor Economics (IZA).
    • Handle: RePEc:iza:izadps:dp15321
    as

    Download full text from publisher

    File URL: https://docs.iza.org/dp15321.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Bruno Crépon & Esther Duflo & Marc Gurgand & Roland Rathelot & Philippe Zamora, 2013. "Do Labor Market Policies have Displacement Effects? Evidence from a Clustered Randomized Experiment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 128(2), pages 531-580.
    2. Lisa Barrow & Lashawn Richburg-Hayes & Cecilia Elena Rouse & Thomas Brock, 2014. "Paying for Performance: The Education Impacts of a Community College Scholarship Program for Low-Income Adults," Journal of Labor Economics, University of Chicago Press, vol. 32(3), pages 563-599.
    3. Betts, Julian R. & Grogger, Jeff, 2003. "The impact of grading standards on student achievement, educational attainment, and entry-level earnings," Economics of Education Review, Elsevier, vol. 22(4), pages 343-352, August.
    4. Philip Oreopoulos & Daniel Lang & Joshua Angrist, 2009. "Incentives and Services for College Achievement: Evidence from a Randomized Trial," American Economic Journal: Applied Economics, American Economic Association, vol. 1(1), pages 136-163, January.
    5. Calsamiglia, Caterina & Loviglio, Annalisa, 2019. "Grading on a curve: When having good peers is not good," Economics of Education Review, Elsevier, vol. 73(C).
    6. Joshua Angrist & Philip Oreopoulos & Tyler Williams, 2014. "When Opportunity Knocks, Who Answers?: New Evidence on College Achievement Awards," Journal of Human Resources, University of Wisconsin Press, vol. 49(3), pages 572-610.
    7. Susan Dynarski & Joshua Hyman & Diane Whitmore Schanzenbach, 2013. "Experimental Evidence on the Effect of Childhood Investments on Postsecondary Attainment and Degree Completion," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 32(4), pages 692-717, September.
    8. Timothy N. Bond & Kevin Lang, 2013. "The Evolution of the Black-White Test Score Gap in Grades K–3: The Fragility of Results," The Review of Economics and Statistics, MIT Press, vol. 95(5), pages 1468-1479, December.
    9. Kristin F. Butcher & Patrick J. McEwan & Akila Weerapana, 2014. "The Effects of an Anti-grade-Inflation Policy at Wellesley College," Journal of Economic Perspectives, American Economic Association, vol. 28(3), pages 189-204, Summer.
    10. David J. Deming & Sarah Cohodes & Jennifer Jennings & Christopher Jencks, 2016. "School Accountability, Postsecondary Attainment, and Earnings," The Review of Economics and Statistics, MIT Press, vol. 98(5), pages 848-862, December.
    11. Tyler Ransom & Esteban Aucejo & Arnaud Maurel & Peter Arcidiacono, 2014. "College Attrition and the Dynamics of Information Revelation," 2014 Meeting Papers 529, Society for Economic Dynamics.
    12. Alison Evans Cuellar & Dhaval M. Dave, 2015. "Causal Effects of Mental Health Treatment on Education Outcomes for Youth in the Justice System," NBER Working Papers 21206, National Bureau of Economic Research, Inc.
    13. Rebecca Diamond & Petra Persson, 2016. "The Long-term Consequences of Teacher Discretion in Grading of High-stakes Tests," NBER Working Papers 22207, National Bureau of Economic Research, Inc.
    14. Philip J. Cook & Kenneth Dodge & George Farkas & Roland G. Fryer, Jr & Jonathan Guryan & Jens Ludwig & Susan Mayer & Harold Pollack & Laurence Steinberg, 2014. "The (Surprising) Efficacy of Academic and Behavioral Intervention with Disadvantaged Youth: Results from a Randomized Experiment in Chicago," NBER Working Papers 19862, National Bureau of Economic Research, Inc.
    15. Scott Carrell & Bruce Sacerdote, 2017. "Why Do College-Going Interventions Work?," American Economic Journal: Applied Economics, American Economic Association, vol. 9(3), pages 124-151, July.
    16. Dave, Dhaval M., 2016. "Causal effects of mental health treatment on education outcomes for youth in the justice systemAuthor-Name: Cuellar, Alison," Economics of Education Review, Elsevier, vol. 54(C), pages 321-339.
    17. Angrist, Joshua D., 2014. "The perils of peer effects," Labour Economics, Elsevier, vol. 30(C), pages 98-108.
    18. Jonathan M.V. Davis & Jonathan Guryan & Kelly Hallberg & Jens Ludwig, 2017. "The Economics of Scale-Up," NBER Working Papers 23925, National Bureau of Economic Research, Inc.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Adam M. Lavecchia & Heidi Liu & Philip Oreopoulos, 2014. "Behavioral Economics of Education: Progress and Possibilities," NBER Working Papers 20609, National Bureau of Economic Research, Inc.
    2. Damgaard, Mette Trier & Nielsen, Helena Skyt, 2018. "Nudging in education," Economics of Education Review, Elsevier, vol. 64(C), pages 313-342.
    3. Scott E. Carrell & Michal Kurlaender, 2023. "My Professor Cares: Experimental Evidence on the Role of Faculty Engagement," American Economic Journal: Economic Policy, American Economic Association, vol. 15(4), pages 113-141, November.
    4. Roland G. Fryer, Jr, 2016. "The Production of Human Capital in Developed Countries: Evidence from 196 Randomized Field Experiments," NBER Working Papers 22130, National Bureau of Economic Research, Inc.
    5. Lisa Barrow & Cecilia Elena Rouse & Amanda McFarland, 2020. "Who Has the Time? Community College Students’ Time-Use Response to Financial Incentives," Atlantic Economic Journal, Springer;International Atlantic Economic Society, vol. 48(1), pages 35-52, March.
    6. Paloyo, Alfredo R. & Rogan, Sally & Siminski, Peter, 2016. "The effect of supplemental instruction on academic performance: An encouragement design experiment," Economics of Education Review, Elsevier, vol. 55(C), pages 57-69.
    7. Serena Canaan & Antoine Deeb & Pierre Mouganie, 2022. "Adviser Value Added and Student Outcomes: Evidence from Randomly Assigned College Advisers," American Economic Journal: Economic Policy, American Economic Association, vol. 14(4), pages 151-191, November.
    8. Anderson, Drew M. & Goldrick-Rab, Sara, 2018. "Aid after enrollment: Impacts of a statewide grant program at public two-year colleges," Economics of Education Review, Elsevier, vol. 67(C), pages 148-157.
    9. Anderson, Drew M., 2020. "When financial aid is scarce: The challenge of allocating college aid where it is needed most," Journal of Public Economics, Elsevier, vol. 190(C).
    10. Åslund, Olof & Engdahl, Mattias, 2018. "The value of earning for learning: Performance bonuses in immigrant language training," Economics of Education Review, Elsevier, vol. 62(C), pages 192-204.
    11. Steven D. Levitt & John A. List & Sally Sadoff, 2016. "The Effect of Performance-Based Incentives on Educational Achievement: Evidence from a Randomized Experiment," NBER Working Papers 22107, National Bureau of Economic Research, Inc.
    12. Conger, Dylan & Turner, Lesley J., 2017. "The effect of price shocks on undocumented students' college attainment and completion," Journal of Public Economics, Elsevier, vol. 148(C), pages 92-114.
    13. Lisa Barrow & Cecilia Elena Rouse, 2018. "Financial Incentives and Educational Investment: The Impact of Performance-based Scholarships on Student Time Use," Education Finance and Policy, MIT Press, vol. 13(4), pages 419-448, Fall.
    14. David J. Deming & Christopher R. Walters, 2017. "The Impact of Price Caps and Spending Cuts on U.S. Postsecondary Attainment," NBER Working Papers 23736, National Bureau of Economic Research, Inc.
    15. Veronica Rattini, 2022. "The Effects of Financial Aid on Graduation and Labor Market Outcomes: New Evidence from Matched Education-Labor Data," CESifo Working Paper Series 10010, CESifo.
    16. Berry, James & Kim, Hyuncheol Bryant & Son, Hyuk Harry, 2022. "When student incentives do not work: Evidence from a field experiment in Malawi," Journal of Development Economics, Elsevier, vol. 158(C).
    17. De Paola, Maria & Scoppa, Vincenzo, 2015. "Procrastination, academic success and the effectiveness of a remedial program," Journal of Economic Behavior & Organization, Elsevier, vol. 115(C), pages 217-236.
    18. Calsamiglia, Caterina & Loviglio, Annalisa, 2019. "Grading on a curve: When having good peers is not good," Economics of Education Review, Elsevier, vol. 73(C).
    19. Ellis, Jimmy R. & Gershenson, Seth, 2016. "LATE for the Meeting: Gender, Peer Advising, and College Success," IZA Discussion Papers 9956, Institute of Labor Economics (IZA).
    20. Joshua D. Angrist & Sarah R. Cohodes & Susan M. Dynarski & Parag A. Pathak & Christopher R. Walters, 2016. "Stand and Deliver: Effects of Boston's Charter High Schools on College Preparation, Entry, and Choice," Journal of Labor Economics, University of Chicago Press, vol. 34(2), pages 275-318.

    More about this item

    Keywords

    program evaluation; grades; curves; gpa; education;
    All these keywords.

    JEL classification:

    • I21 - Health, Education, and Welfare - - Education - - - Analysis of Education
    • I26 - Health, Education, and Welfare - - Education - - - Returns to Education
    • C21 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Cross-Sectional Models; Spatial Models; Treatment Effect Models

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:iza:izadps:dp15321. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Holger Hinte (email available below). General contact details of provider: https://edirc.repec.org/data/izaaade.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.