IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/18038.html
   My bibliography  Save this paper

Knowledge, Tests, and Fadeout in Educational Interventions

Author

Listed:
  • Elizabeth U. Cascio
  • Douglas O. Staiger

Abstract

Educational interventions are often evaluated and compared on the basis of their impacts on test scores. Decades of research have produced two empirical regularities: interventions in later grades tend to have smaller effects than the same interventions in earlier grades, and the test score impacts of early educational interventions almost universally "fade out" over time. This paper explores whether these empirical regularities are an artifact of the common practice of rescaling test scores in terms of a student's position in a widening distribution of knowledge. If a standard deviation in test scores in later grades translates into a larger difference in knowledge, an intervention's effect on normalized test scores may fall even as its effect on knowledge does not. We evaluate this hypothesis by fitting a model of education production to correlations in test scores across grades and with college-going using both administrative and survey data. Our results imply that the variance in knowledge does indeed rise as children progress through school, but not enough for test score normalization to fully explain these empirical regularities.

Suggested Citation

  • Elizabeth U. Cascio & Douglas O. Staiger, 2012. "Knowledge, Tests, and Fadeout in Educational Interventions," NBER Working Papers 18038, National Bureau of Economic Research, Inc.
  • Handle: RePEc:nbr:nberwo:18038
    Note: CH ED LS
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w18038.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Jesse Rothstein, 2010. "Teacher Quality in Educational Production: Tracking, Decay, and Student Achievement," The Quarterly Journal of Economics, Oxford University Press, vol. 125(1), pages 175-214.
    2. Thomas J. Kane & Douglas O. Staiger, 2008. "Estimating Teacher Impacts on Student Achievement: An Experimental Evaluation," NBER Working Papers 14607, National Bureau of Economic Research, Inc.
    3. Raj Chetty & John N. Friedman & Nathaniel Hilger & Emmanuel Saez & Diane Whitmore Schanzenbach & Danny Yagan, 2011. "How Does Your Kindergarten Classroom Affect Your Earnings? Evidence from Project Star," The Quarterly Journal of Economics, Oxford University Press, vol. 126(4), pages 1593-1660.
    4. Timothy N. Bond & Kevin Lang, 2013. "The Evolution of the Black-White Test Score Gap in Grades K–3: The Fragility of Results," The Review of Economics and Statistics, MIT Press, vol. 95(5), pages 1468-1479, December.
    5. Kevin Lang, 2010. "Measurement Matters: Perspectives on Education Policy from an Economist and School Board Member," Journal of Economic Perspectives, American Economic Association, vol. 24(3), pages 167-182, Summer.
    6. Fryer Jr., Roland G., 2011. "Racial Inequality in the 21st Century: The Declining Significance of Discrimination," Handbook of Labor Economics, in: O. Ashenfelter & D. Card (ed.), Handbook of Labor Economics, edition 1, volume 4, chapter 10, pages 855-971, Elsevier.
    7. Flavio Cunha & James J. Heckman, 2008. "Formulating, Identifying and Estimating the Technology of Cognitive and Noncognitive Skill Formation," Journal of Human Resources, University of Wisconsin Press, vol. 43(4).
    8. Kane, Thomas J. & Rockoff, Jonah E. & Staiger, Douglas O., 2008. "What does certification tell us about teacher effectiveness? Evidence from New York City," Economics of Education Review, Elsevier, vol. 27(6), pages 615-631, December.
    9. Neal, Derek A & Johnson, William R, 1996. "The Role of Premarket Factors in Black-White Wage Differences," Journal of Political Economy, University of Chicago Press, vol. 104(5), pages 869-895, October.
    10. Krueger, Alan B & Whitmore, Diane M, 2001. "The Effect of Attending a Small Class in the Early Grades on College-Test Taking and Middle School Test Results: Evidence from Project STAR," Economic Journal, Royal Economic Society, vol. 111(468), pages 1-28, January.
    11. Will Dobbie & Roland G. Fryer, 2011. "Are High-Quality Schools Enough to Increase Achievement among the Poor? Evidence from the Harlem Children's Zone," American Economic Journal: Applied Economics, American Economic Association, vol. 3(3), pages 158-187, July.
    12. Raj Chetty & John N. Friedman & Jonah E. Rockoff, 2011. "The Long-Term Impacts of Teachers: Teacher Value-Added and Student Outcomes in Adulthood," NBER Working Papers 17699, National Bureau of Economic Research, Inc.
    13. Elizabeth U. Cascio & Ethan G. Lewis, 2006. "Schooling and the Armed Forces Qualifying Test: Evidence from School-Entry Laws," Journal of Human Resources, University of Wisconsin Press, vol. 41(2).
    14. Daniel F. McCaffrey & J. R. Lockwood & Daniel Koretz & Thomas A. Louis & Laura Hamilton, 2004. "Models for Value-Added Modeling of Teacher Effects," Journal of Educational and Behavioral Statistics, , vol. 29(1), pages 67-101, March.
    15. John Cawley & James Heckman & Edward Vytlacil, 1999. "On Policies To Reward The Value Added By Educators," The Review of Economics and Statistics, MIT Press, vol. 81(4), pages 720-727, November.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Sean Corcoran & Dan Goldhaber, 2013. "Value Added and Its Uses: Where You Stand Depends on Where You Sit," Education Finance and Policy, MIT Press, vol. 8(3), pages 418-434, July.
    2. repec:hrv:faseco:30749606 is not listed on IDEAS
    3. Shaun M. Dougherty, 2015. "Bridging the Discontinuity in Adolescent Literacy? Mixed Evidence from a Middle Grades Intervention," Education Finance and Policy, MIT Press, vol. 10(2), pages 157-192, March.
    4. Philip J. Cook & Kenneth Dodge & George Farkas & Roland G. Fryer, Jr & Jonathan Guryan & Jens Ludwig & Susan Mayer & Harold Pollack & Laurence Steinberg, 2014. "The (Surprising) Efficacy of Academic and Behavioral Intervention with Disadvantaged Youth: Results from a Randomized Experiment in Chicago," NBER Working Papers 19862, National Bureau of Economic Research, Inc.
    5. Patrick Kline & Christopher R. Walters, 2016. "Evaluating Public Programs with Close Substitutes: The Case of HeadStart," The Quarterly Journal of Economics, Oxford University Press, vol. 131(4), pages 1795-1848.
    6. Monique De Haan & Edwin Leuven, 2020. "Head Start and the Distribution of Long-Term Education and Labor Market Outcomes," Journal of Labor Economics, University of Chicago Press, vol. 38(3), pages 727-765.
    7. C. Kirabo Jackson, 2018. "What Do Test Scores Miss? The Importance of Teacher Effects on Non–Test Score Outcomes," Journal of Political Economy, University of Chicago Press, vol. 126(5), pages 2072-2107.
    8. Jo Blanden & Emilia Del Bono & Sandra McNally & Birgitta Rabe, 2016. "Universal Pre‐school Education: The Case of Public Funding with Private Provision," Economic Journal, Royal Economic Society, vol. 0(592), pages 682-723, May.
    9. Duflo, Esther & Dupas, Pascaline & Kremer, Michael, 2015. "School governance, teacher incentives, and pupil–teacher ratios: Experimental evidence from Kenyan primary schools," Journal of Public Economics, Elsevier, vol. 123(C), pages 92-110.
    10. Raj Chetty & John N. Friedman & Jonah E. Rockoff, 2014. "Measuring the Impacts of Teachers II: Teacher Value-Added and Student Outcomes in Adulthood," American Economic Review, American Economic Association, vol. 104(9), pages 2633-2679, September.
    11. Brian Jacob & Jesse Rothstein, 2016. "The Measurement of Student Ability in Modern Assessment Systems," Journal of Economic Perspectives, American Economic Association, vol. 30(3), pages 85-108, Summer.
    12. Wan, Sirui & Bond, Timothy N. & Lang, Kevin & Clements, Douglas H. & Sarama, Julie & Bailey, Drew H., 2021. "Is intervention fadeout a scaling artefact?," Economics of Education Review, Elsevier, vol. 82(C).
    13. Carsten Schroeder & Shlomo Yitzhaki, 2020. "Exploring the robustness of country rankings by educational attainment," Journal of Economics, Springer, vol. 129(3), pages 271-296, April.
    14. Sashwati Banerjee & Sujoy Chakravarty & Ira Joshi & Siddharth Pillai, 2018. "Can Digital Technologies Play a Role in Improving Children’s Learning Outcomes in India?," Journal of Development Policy and Practice, , vol. 3(1), pages 55-86, January.
    15. Schwerdt, Guido & West, Martin R. & Winters, Marcus A., 2017. "The effects of test-based retention on student outcomes over time: Regression discontinuity evidence from Florida," Journal of Public Economics, Elsevier, vol. 152(C), pages 154-169.
    16. Elizabeth U. Cascio & Diane Whitmore Schanzenbach, 2016. "First in the Class? Age and the Education Production Function," Education Finance and Policy, MIT Press, vol. 11(3), pages 225-250, Summer.
    17. C. Kirabo Jackson, 2012. "Non-Cognitive Ability, Test Scores, and Teacher Quality: Evidence from 9th Grade Teachers in North Carolina," NBER Working Papers 18624, National Bureau of Economic Research, Inc.
    18. Jeffrey Penney, 2017. "Racial Interaction Effects and Student Achievement," Education Finance and Policy, MIT Press, vol. 12(4), pages 447-467, Fall.
    19. Backes, Ben & Cowan, James, 2019. "Is the pen mightier than the keyboard? The effect of online testing on measured student achievement," Economics of Education Review, Elsevier, vol. 68(C), pages 89-103.
    20. Douglas Almond & Bhashkar Mazumder & Reyn Ewijk, 2015. "In Utero Ramadan Exposure and Children's Academic Performance," Economic Journal, Royal Economic Society, vol. 125(589), pages 1501-1533, December.
    21. de Ree, Joppe & Maggioni, Mario A. & Paulle, Bowen & Rossignoli, Domenico & Ruijs, Nienke & Walentek, Dawid, 2021. "Closing the income-achievement gap? Experimental evidence from high-dosage tutoring in Dutch primary education," SocArXiv qepc2, Center for Open Science.
    22. Julie Berry Cullen & Steven D. Levitt & Erin Robertson & Sally Sadoff, 2013. "What Can Be Done to Improve Struggling High Schools?," Journal of Economic Perspectives, American Economic Association, vol. 27(2), pages 133-152, Spring.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Joshua D. Angrist & Sarah R. Cohodes & Susan M. Dynarski & Parag A. Pathak & Christopher R. Walters, 2016. "Stand and Deliver: Effects of Boston's Charter High Schools on College Preparation, Entry, and Choice," Journal of Labor Economics, University of Chicago Press, vol. 34(2), pages 275-318.
    2. Raj Chetty & John N. Friedman & Nathaniel Hilger & Emmanuel Saez & Diane Whitmore Schanzenbach & Danny Yagan, 2011. "How Does Your Kindergarten Classroom Affect Your Earnings? Evidence from Project Star," The Quarterly Journal of Economics, Oxford University Press, vol. 126(4), pages 1593-1660.
    3. Nirav Mehta, 2019. "Measuring quality for use in incentive schemes: The case of “shrinkage” estimators," Quantitative Economics, Econometric Society, vol. 10(4), pages 1537-1577, November.
    4. Koedel, Cory & Mihaly, Kata & Rockoff, Jonah E., 2015. "Value-added modeling: A review," Economics of Education Review, Elsevier, vol. 47(C), pages 180-195.
    5. Richard J. Murnane, 2013. "U.S. High School Graduation Rates: Patterns and Explanations," Journal of Economic Literature, American Economic Association, vol. 51(2), pages 370-422, June.
    6. Dan Goldhaber & Michael Hansen, 2013. "Is it Just a Bad Class? Assessing the Long-term Stability of Estimated Teacher Performance," Economica, London School of Economics and Political Science, vol. 80(319), pages 589-612, July.
    7. Will Dobbie & Roland G. Fryer, 2011. "Are High-Quality Schools Enough to Increase Achievement among the Poor? Evidence from the Harlem Children's Zone," American Economic Journal: Applied Economics, American Economic Association, vol. 3(3), pages 158-187, July.
    8. Mariesa A. Herrmann & Jonah E. Rockoff, 2012. "Worker Absence and Productivity: Evidence from Teaching," Journal of Labor Economics, University of Chicago Press, vol. 30(4), pages 749-782.
    9. Seth Gershenson & Diane Whitmore Schanzenbach, 2016. "Linking Teacher Quality, Student Attendance, and Student Achievement," Education Finance and Policy, MIT Press, vol. 11(2), pages 125-149, Spring.
    10. Tahir Andrabi & Jishnu Das & Asim Ijaz Khwaja & Tristan Zajonc, 2011. "Do Value-Added Estimates Add Value? Accounting for Learning Dynamics," American Economic Journal: Applied Economics, American Economic Association, vol. 3(3), pages 29-54, July.
    11. Koedel Cory & Leatherman Rebecca & Parsons Eric, 2012. "Test Measurement Error and Inference from Value-Added Models," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 12(1), pages 1-37, November.
    12. Bhalotra, Sonia R. & Karlsson, Martin & Nilsson, Therese & Schwarz, Nina, 2016. "Infant Health, Cognitive Performance and Earnings: Evidence from Inception of the Welfare State in Sweden," IZA Discussion Papers 10339, Institute of Labor Economics (IZA).
    13. Allison Atteberry & Susanna Loeb & James Wyckoff, 2013. "Do First Impressions Matter? Improvement in Early Career Teacher Effectiveness," NBER Working Papers 19096, National Bureau of Economic Research, Inc.
    14. Stacy, Brian & Guarino, Cassandra & Wooldridge, Jeffrey, 2018. "Does the precision and stability of value-added estimates of teacher performance depend on the types of students they serve?," Economics of Education Review, Elsevier, vol. 64(C), pages 50-74.
    15. repec:mpr:mprres:8135 is not listed on IDEAS
    16. Vilsa E. Curto & Roland G. Fryer Jr., 2014. "The Potential of Urban Boarding Schools for the Poor: Evidence from SEED," Journal of Labor Economics, University of Chicago Press, vol. 32(1), pages 65-93.
    17. C. Kirabo Jackson, 2012. "Non-Cognitive Ability, Test Scores, and Teacher Quality: Evidence from 9th Grade Teachers in North Carolina," NBER Working Papers 18624, National Bureau of Economic Research, Inc.
    18. Will Dobbie & Roland G. Fryer, Jr, 2009. "Are High Quality Schools Enough to Close the Achievement Gap? Evidence from a Social Experiment in Harlem," NBER Working Papers 15473, National Bureau of Economic Research, Inc.
    19. Douglas Almond & Bhashkar Mazumder & Reyn Ewijk, 2015. "In Utero Ramadan Exposure and Children's Academic Performance," Economic Journal, Royal Economic Society, vol. 125(589), pages 1501-1533, December.
    20. Roland G. Fryer, Jr, 2016. "The Production of Human Capital in Developed Countries: Evidence from 196 Randomized Field Experiments," NBER Working Papers 22130, National Bureau of Economic Research, Inc.
    21. Douglas Almond & Bhashkar Mazumder & Reyn van Ewijk, 2011. "Fasting During Pregnancy and Children's Academic Performance," NBER Working Papers 17713, National Bureau of Economic Research, Inc.

    More about this item

    JEL classification:

    • I20 - Health, Education, and Welfare - - Education - - - General
    • I21 - Health, Education, and Welfare - - Education - - - Analysis of Education
    • I28 - Health, Education, and Welfare - - Education - - - Government Policy
    • J24 - Labor and Demographic Economics - - Demand and Supply of Labor - - - Human Capital; Skills; Occupational Choice; Labor Productivity

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:18038. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: . General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.