IDEAS home Printed from https://ideas.repec.org/p/ecl/harjfk/rwp09-034.html
   My bibliography  Save this paper

Do Value-Added Estimates Add Value? Accounting for Learning Dynamics

Author

Listed:
  • Andrabi, Tahir

    (Pomona College)

  • Das, Jishnu

    (World Bank)

  • Khwaja, Asim Ijaz

    (Harvard University)

  • Zajonc, Tristan

    (Harvard University)

Abstract

Evaluations of educational programs commonly assume that what children learn persists over time. The authors compare learning in Pakistani public and private schools using dynamic panel methods that account for three key empirical challenges to widely used value-added models: imperfect persistence, unobserved student heterogeneity, and measurement error. Their estimates suggest that only a fifth to a half of learning persists between grades and that private schools increase average achievement by 0.25 standard deviations each year. In contrast, estimates from commonly used value-added models significantly understate the impact of private schools' on student achievement and/or overstate persistence. These results have implications for program evaluation and value-added accountability system design.

Suggested Citation

  • Andrabi, Tahir & Das, Jishnu & Khwaja, Asim Ijaz & Zajonc, Tristan, 2009. "Do Value-Added Estimates Add Value? Accounting for Learning Dynamics," Working Paper Series rwp09-034, Harvard University, John F. Kennedy School of Government.
  • Handle: RePEc:ecl:harjfk:rwp09-034
    as

    Download full text from publisher

    File URL: https://research.hks.harvard.edu/publications/workingpapers/citation.aspx?PubId=6872&type=WPN
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Alok Bhargava & J. D. Sargan, 2006. "Estimating Dynamic Random Effects Models From Panel Data Covering Short Time Periods," World Scientific Book Chapters, in: Econometrics, Statistics And Computational Approaches In Food And Health Sciences, chapter 1, pages 3-27, World Scientific Publishing Co. Pte. Ltd..
    2. Jesse Rothstein, 2010. "Teacher Quality in Educational Production: Tracking, Decay, and Student Achievement," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 125(1), pages 175-214.
    3. David Deming, 2009. "Early Childhood Intervention and Life-Cycle Skill Development: Evidence from Head Start," American Economic Journal: Applied Economics, American Economic Association, vol. 1(3), pages 111-134, July.
    4. Abhijit V. Banerjee & Shawn Cole & Esther Duflo & Leigh Linden, 2007. "Remedying Education: Evidence from Two Randomized Experiments in India," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(3), pages 1235-1264.
    5. Tahir Andrabi & Jishnu Das & Asim Ijaz Khwaja & Tristan Zajonc, 2011. "Do Value-Added Estimates Add Value? Accounting for Learning Dynamics," American Economic Journal: Applied Economics, American Economic Association, vol. 3(3), pages 29-54, July.
    6. Joseph G. Altonji & Todd E. Elder & Christopher R. Taber, 2005. "Selection on Observed and Unobserved Variables: Assessing the Effectiveness of Catholic Schools," Journal of Political Economy, University of Chicago Press, vol. 113(1), pages 151-184, February.
    7. Arellano, Manuel & Bover, Olympia, 1995. "Another look at the instrumental variable estimation of error-components models," Journal of Econometrics, Elsevier, vol. 68(1), pages 29-51, July.
    8. James Heckman & Flavio Cunha, 2007. "The Technology of Skill Formation," American Economic Review, American Economic Association, vol. 97(2), pages 31-47, May.
    9. Thomas J. Kane & Douglas O. Staiger, 2008. "Estimating Teacher Impacts on Student Achievement: An Experimental Evaluation," NBER Working Papers 14607, National Bureau of Economic Research, Inc.
    10. Jesse Rothstein, 2007. "Do Value-Added Models Add Value? Tracking, Fixed Effects, and Causal Inference," Working Papers 1036, Princeton University, Department of Economics, Center for Economic Policy Studies..
    11. Thomas J. Kane & Douglas O. Staiger, 2002. "The Promise and Pitfalls of Using Imprecise School Accountability Measures," Journal of Economic Perspectives, American Economic Association, vol. 16(4), pages 91-114, Fall.
    12. Santibanez, Lucrecia, 2006. "Why we should care if teachers get A's: Teacher test scores and student achievement in Mexico," Economics of Education Review, Elsevier, vol. 25(5), pages 510-520, October.
    13. Michael Kremer & Edward Miguel & Rebecca Thornton, 2009. "Incentives to Learn," The Review of Economics and Statistics, MIT Press, vol. 91(3), pages 437-456, August.
    14. Brian A. Jacob & Steven D. Levitt, 2003. "Rotten Apples: An Investigation of the Prevalence and Predictors of Teacher Cheating," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 118(3), pages 843-877.
    15. Arellano, Manuel & Honore, Bo, 2001. "Panel data models: some recent developments," Handbook of Econometrics, in: J.J. Heckman & E.E. Leamer (ed.), Handbook of Econometrics, edition 1, volume 5, chapter 53, pages 3229-3296, Elsevier.
    16. Jesse Rothstein, 2007. "Do Value-Added Models Add Value? Tracking, Fixed Effects, and Causal Inference," Working Papers 1036, Princeton University, Department of Economics, Center for Economic Policy Studies..
    17. Kenneth Y. Chay & Patrick J. McEwan & Miguel Urquiola, 2005. "The Central Role of Noise in Evaluating Interventions That Use Test Scores to Rank Schools," American Economic Review, American Economic Association, vol. 95(4), pages 1237-1258, September.
    18. Harold Alderman & Peter F. Orazem & Elizabeth M. Paterno, 2001. "School Quality, School Cost, and the Public/Private School Choices of Low-Income Households in Pakistan," Journal of Human Resources, University of Wisconsin Press, vol. 36(2), pages 304-326.
    19. Paul Glewwe & Nauman Ilias & Michael Kremer, 2010. "Teacher Incentives," American Economic Journal: Applied Economics, American Economic Association, vol. 2(3), pages 205-227, July.
    20. Sass, Tim R. & Semykina, Anastasia & Harris, Douglas N., 2014. "Value-added models and the measurement of teacher productivity," Economics of Education Review, Elsevier, vol. 38(C), pages 9-23.
    21. Eric A. Hanushek, 1979. "Conceptual and Empirical Issues in the Estimation of Educational Production Functions," Journal of Human Resources, University of Wisconsin Press, vol. 14(3), pages 351-388.
    22. Blundell, Richard & Bond, Stephen, 1998. "Initial conditions and moment restrictions in dynamic panel data models," Journal of Econometrics, Elsevier, vol. 87(1), pages 115-143, August.
    23. Currie, Janet & Thomas, Duncan, 1995. "Does Head Start Make a Difference?," American Economic Review, American Economic Association, vol. 85(3), pages 341-364, June.
    24. Flavio Cunha & James J. Heckman & Susanne M. Schennach, 2010. "Estimating the Technology of Cognitive and Noncognitive Skill Formation," Econometrica, Econometric Society, vol. 78(3), pages 883-931, May.
    25. Joshua Angrist & Eric Bettinger & Erik Bloom & Elizabeth King & Michael Kremer, 2002. "Vouchers for Private Schooling in Colombia: Evidence from a Randomized Natural Experiment," American Economic Review, American Economic Association, vol. 92(5), pages 1535-1558, December.
    26. Neal, Derek A & Johnson, William R, 1996. "The Role of Premarket Factors in Black-White Wage Differences," Journal of Political Economy, University of Chicago Press, vol. 104(5), pages 869-895, October.
    27. Petra E. Todd & Kenneth I. Wolpin, 2003. "On The Specification and Estimation of The Production Function for Cognitive Achievement," Economic Journal, Royal Economic Society, vol. 113(485), pages 3-33, February.
    28. Jacob, Brian A. & Lefgren, Lars, 2005. "What Do Parents Value in Education? An Investigation of Parents' Revealed Preferences for Teachers," Working Paper Series rwp05-043, Harvard University, John F. Kennedy School of Government.
    29. Krueger, Alan B & Whitmore, Diane M, 2001. "The Effect of Attending a Small Class in the Early Grades on College-Test Taking and Middle School Test Results: Evidence from Project STAR," Economic Journal, Royal Economic Society, vol. 111(468), pages 1-28, January.
    30. James J. Heckman & Dimitriy V. Masterov, 2007. "The Productivity Argument for Investing in Young Children," Review of Agricultural Economics, Agricultural and Applied Economics Association, vol. 29(3), pages 446-493.
    31. Murnane, Richard J & Willett, John B & Levy, Frank, 1995. "The Growing Importance of Cognitive Skills in Wage Determination," The Review of Economics and Statistics, MIT Press, vol. 77(2), pages 251-266, May.
    32. Alan B. Krueger, 2003. "Economic Considerations and Class Size," Economic Journal, Royal Economic Society, vol. 113(485), pages 34-63, February.
    33. Arellano, Manuel, 2003. "Panel Data Econometrics," OUP Catalogue, Oxford University Press, number 9780199245291.
    34. Tim R. Sass, 2006. "Charter Schools and Student Achievement in Florida," Education Finance and Policy, MIT Press, vol. 1(1), pages 91-122, January.
    35. Alderman, Harold & Kim, Jooseop & Orazem, Peter F., 2003. "Design, evaluation, and sustainability of private schools for the poor: the Pakistan urban and rural fellowship school experiments," Economics of Education Review, Elsevier, vol. 22(3), pages 265-274, June.
    36. Andrabi, Tahir & Das, Jishnu & Khwaja, Asim Ijaz, 2006. "A dime a day : the possibilities and limits of private schooling in Pakistan," Policy Research Working Paper Series 4066, The World Bank.
    37. Jimenez, Emmanuel & Lockheed, Marlaine E & Paqueo, Vicente, 1991. "The Relative Efficiency of Private and Public Schools in Developing Countries," The World Bank Research Observer, World Bank, vol. 6(2), pages 205-218, July.
    38. Eric A. Hanushek, 2003. "The Failure of Input-Based Schooling Policies," Economic Journal, Royal Economic Society, vol. 113(485), pages 64-98, February.
    39. Melissa Osborne & Herbert Gintis & Samuel Bowles, 2001. "The Determinants of Earnings: A Behavioral Approach," Journal of Economic Literature, American Economic Association, vol. 39(4), pages 1137-1176, December.
    40. Brian A. Jacob & Lars Lefgren, 2007. "What Do Parents Value in Education? An Empirical Investigation of Parents' Revealed Preferences for Teachers," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 122(4), pages 1603-1637.
    41. repec:pri:edures:25ers.pdf is not listed on IDEAS
    42. Brian A. Jacob & Lars Lefgren & David Sims, 2008. "The Persistence of Teacher-Induced Learning Gains," NBER Working Papers 14065, National Bureau of Economic Research, Inc.
    43. Ladd, Helen F. & Walsh, Randall P., 2002. "Implementing value-added measures of school effectiveness: getting the incentives right," Economics of Education Review, Elsevier, vol. 21(1), pages 1-17, February.
    44. Abowd J.M. & Crepon B. & Kramarz F., 2001. "Moment Estimation With Attrition: An Application to Economic Models," Journal of the American Statistical Association, American Statistical Association, vol. 96, pages 1223-1231, December.
    45. Manuel Arellano & Stephen Bond, 1991. "Some Tests of Specification for Panel Data: Monte Carlo Evidence and an Application to Employment Equations," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 58(2), pages 277-297.
    46. Daniel G. Sullivan, 2001. "A note on the estimation of linear regression models with Heteroskedastic measurement errors," Working Paper Series WP-01-23, Federal Reserve Bank of Chicago.
    47. Yona Rubinstein & James J. Heckman, 2001. "The Importance of Noncognitive Skills: Lessons from the GED Testing Program," American Economic Review, American Economic Association, vol. 91(2), pages 145-149, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Andrabi, Tahir & Das, Jishnu & Khwaja, Asim Ijaz & Zajonc, Tristan, 2009. "Here Today, Gone Tomorrow? Examining the Extent and Implications of Low Persistence in Child Learning," Scholarly Articles 4412571, Harvard Kennedy School of Government.
    2. Marine de Talancé, 2015. "Better Teachers, Better Results? Evidence from Rural Pakistan," Working Papers DT/2015/21, DIAL (Développement, Institutions et Mondialisation).
    3. Brian A. Jacob & Lars Lefgren & David Sims, 2008. "The Persistence of Teacher-Induced Learning Gains," NBER Working Papers 14065, National Bureau of Economic Research, Inc.
    4. Raj Chetty & John N. Friedman & Nathaniel Hilger & Emmanuel Saez & Diane Whitmore Schanzenbach & Danny Yagan, 2011. "How Does Your Kindergarten Classroom Affect Your Earnings? Evidence from Project Star," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 126(4), pages 1593-1660.
    5. Emilia Del Bono & Marco Francesconi & Yvonne Kelly & Amanda Sacker, 2016. "Early Maternal Time Investment and Early Child Outcomes," Economic Journal, Royal Economic Society, vol. 126(596), pages 96-135, October.
    6. Koedel, Cory & Mihaly, Kata & Rockoff, Jonah E., 2015. "Value-added modeling: A review," Economics of Education Review, Elsevier, vol. 47(C), pages 180-195.
    7. Godstime Osekhebhen Eigbiremolen, 2020. "Estimating Private School Premium for Primary School Children in Ethiopia: Evidence from Individual-level Panel Data," Progress in Development Studies, , vol. 20(1), pages 26-44, January.
    8. Nicholas W Papageorge & Kevin Thom, 2020. "Genes, Education, and Labor Market Outcomes: Evidence from the Health and Retirement Study," Journal of the European Economic Association, European Economic Association, vol. 18(3), pages 1351-1399.
    9. Jesse Rothstein, 2010. "Teacher Quality in Educational Production: Tracking, Decay, and Student Achievement," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 125(1), pages 175-214.
    10. Elke Lüdemann, 2011. "Schooling and the Formation of Cognitive and Non-cognitive Outcomes," ifo Beiträge zur Wirtschaftsforschung, ifo Institute - Leibniz Institute for Economic Research at the University of Munich, number 39.
    11. Cassandra M. Guarino & Mark D. Reckase & Jeffrey M. Woolrdige, 2014. "Can Value-Added Measures of Teacher Performance Be Trusted?," Education Finance and Policy, MIT Press, vol. 10(1), pages 117-156, November.
    12. Singh, Abhijeet, 2015. "Private school effects in urban and rural India: Panel estimates at primary and secondary school ages," Journal of Development Economics, Elsevier, vol. 113(C), pages 16-32.
    13. Miguel Urquiola, 2015. "Progress and challenges in achieving an evidence-based education policy in Latin America and the Caribbean," Latin American Economic Review, Springer;Centro de Investigaciòn y Docencia Económica (CIDE), vol. 24(1), pages 1-30, December.
    14. Ding, Weili & Lehrer, Steven F., 2014. "Understanding the role of time-varying unobserved ability heterogeneity in education production," Economics of Education Review, Elsevier, vol. 40(C), pages 55-75.
    15. Zhang, Hongliang & Behrman, Jere R. & Fan, C. Simon & Wei, Xiangdong & Zhang, Junsen, 2014. "Does parental absence reduce cognitive achievements? Evidence from rural China," Journal of Development Economics, Elsevier, vol. 111(C), pages 181-195.
    16. Koedel Cory & Leatherman Rebecca & Parsons Eric, 2012. "Test Measurement Error and Inference from Value-Added Models," The B.E. Journal of Economic Analysis & Policy, De Gruyter, vol. 12(1), pages 1-37, November.
    17. Joshua D. Angrist & Sarah R. Cohodes & Susan M. Dynarski & Parag A. Pathak & Christopher R. Walters, 2016. "Stand and Deliver: Effects of Boston's Charter High Schools on College Preparation, Entry, and Choice," Journal of Labor Economics, University of Chicago Press, vol. 34(2), pages 275-318.
    18. Kautz, Tim & Heckman, James J. & Diris, Ron & ter Weel, Bas & Borghans, Lex, 2014. "Fostering and Measuring Skills: Improving Cognitive and Non-Cognitive Skills to Promote Lifetime Success," IZA Discussion Papers 8696, Institute of Labor Economics (IZA).
    19. Cobb-Clark Deborah A. & Harmon Colm & Staneva Anita, 2021. "The bilingual gap in children's language, emotional, and pro-social development," IZA Journal of Labor Economics, Sciendo & Forschungsinstitut zur Zukunft der Arbeit GmbH (IZA), vol. 10(1), pages 1-41, January.
    20. Greta Morando & Lucinda Platt, 2022. "The Impact of Centre‐based Childcare on Non‐cognitive Skills of Young Children," Economica, London School of Economics and Political Science, vol. 89(356), pages 908-946, October.

    More about this item

    JEL classification:

    • I2 - Health, Education, and Welfare - - Education
    • C52 - Mathematical and Quantitative Methods - - Econometric Modeling - - - Model Evaluation, Validation, and Selection
    • C23 - Mathematical and Quantitative Methods - - Single Equation Models; Single Variables - - - Models with Panel Data; Spatio-temporal Models

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:ecl:harjfk:rwp09-034. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/ksharus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.