IDEAS home Printed from https://ideas.repec.org/p/nbr/nberwo/23478.html
   My bibliography  Save this paper

An Evaluation of Bias in Three Measures of Teacher Quality: Value-Added, Classroom Observations, and Student Surveys

Author

Listed:
  • Andrew Bacher-Hicks
  • Mark J. Chin
  • Thomas J. Kane
  • Douglas O. Staiger

Abstract

There are three primary measures of teaching performance: student test-based measures (i.e., value added), classroom observations, and student surveys. Although all three types of measures could be biased by unmeasured traits of the students in teachers’ classrooms, prior research has largely focused on the validity of value-added measures. We conduct an experiment involving 66 mathematics teachers in four school districts and test the validity of all three types of measures. Specifically, we test whether a teacher’s performance on each measure under naturally occurring (i.e., non-experimental) settings predicts performance following random assignment of that teacher to a class of students. Combining our results with those from two previous experiments, we provide further evidence that value-added measures are unbiased predictors of teacher performance. In addition, we provide the first evidence that classroom observation scores are unbiased predictors of teacher performance on a rubric measuring the quality of mathematics instruction. Unfortunately, we lack the statistical power to reach any similar conclusions regarding the predictive validity of a teacher’s student survey responses.

Suggested Citation

  • Andrew Bacher-Hicks & Mark J. Chin & Thomas J. Kane & Douglas O. Staiger, 2017. "An Evaluation of Bias in Three Measures of Teacher Quality: Value-Added, Classroom Observations, and Student Surveys," NBER Working Papers 23478, National Bureau of Economic Research, Inc.
    • Handle: RePEc:nbr:nberwo:23478
    Note: CH ED LS PE
    as

    Download full text from publisher

    File URL: http://www.nber.org/papers/w23478.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. M. Caridad Araujo & Pedro Carneiro & Yyannú Cruz-Aguayo & Norbert Schady, 2016. "Teacher Quality and Learning Outcomes in Kindergarten," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 131(3), pages 1415-1453.
    2. Jesse Rothstein, 2010. "Teacher Quality in Educational Production: Tracking, Decay, and Student Achievement," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 125(1), pages 175-214.
    3. A. Colin Cameron & Jonah B. Gelbach & Douglas L. Miller, 2008. "Bootstrap-Based Improvements for Inference with Clustered Errors," The Review of Economics and Statistics, MIT Press, vol. 90(3), pages 414-427, August.
    4. Thomas J. Kane & Douglas O. Staiger, 2008. "Estimating Teacher Impacts on Student Achievement: An Experimental Evaluation," NBER Working Papers 14607, National Bureau of Economic Research, Inc.
    5. Raj Chetty & John N. Friedman & Jonah E. Rockoff, 2014. "Measuring the Impacts of Teachers I: Evaluating Bias in Teacher Value-Added Estimates," American Economic Review, American Economic Association, vol. 104(9), pages 2593-2632, September.
    6. M. Caridad Araujo & Pedro Carneiro & Yyannú Cruz-Aguayo & Norbert Schady, 2016. "Teacher Quality and Learning Outcomes in Kindergarten," The Quarterly Journal of Economics, Oxford University Press, vol. 131(3), pages 1415-1453.
    7. Kane, Thomas J. & Rockoff, Jonah E. & Staiger, Douglas O., 2008. "What does certification tell us about teacher effectiveness? Evidence from New York City," Economics of Education Review, Elsevier, vol. 27(6), pages 615-631, December.
    8. Andrew Bacher-Hicks & Thomas J. Kane & Douglas O. Staiger, 2014. "Validating Teacher Effect Estimates Using Changes in Teacher Assignments in Los Angeles," NBER Working Papers 20657, National Bureau of Economic Research, Inc.
    9. Jonah E. Rockoff, 2004. "The Impact of Individual Teachers on Student Achievement: Evidence from Panel Data," American Economic Review, American Economic Association, vol. 94(2), pages 247-252, May.
    10. Brian A. Jacob & Lars Lefgren, 2005. "Principals as Agents: Subjective Performance Measurement in Education," NBER Working Papers 11463, National Bureau of Economic Research, Inc.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Filmer,Deon P. & Nahata,Vatsal & Sabarwal,Shwetlena, 2021. "Preparation, Practice, and Beliefs : A Machine Learning Approach to Understanding Teacher Effectiveness," Policy Research Working Paper Series 9847, The World Bank.
    2. Briole, Simon & Maurin, Eric, 2019. "Does Evaluating Teachers Make a Difference?," IZA Discussion Papers 12307, Institute of Labor Economics (IZA).
    3. Esteban M. Aucejo & Patrick Coate & Jane Cooley Fruehwirth & Sean Kelly & Zachary Mozenter, 2018. "Teacher effectiveness and classroom composition," CEP Discussion Papers dp1574, Centre for Economic Performance, LSE.
    4. Jacob, Brian A. & Rockoff, Jonah E. & Taylor, Eric S. & Lindy, Benjamin & Rosen, Rachel, 2018. "Teacher applicant hiring and teacher performance: Evidence from DC public schools," Journal of Public Economics, Elsevier, vol. 166(C), pages 81-97.
    5. David Blazar, 2018. "Validating Teacher Effects on Students’ Attitudes and Behaviors: Evidence from Random Assignment of Teachers to Students," Education Finance and Policy, MIT Press, vol. 13(3), pages 281-309, Summer.
    6. Asma Benhenda, 2018. "Teacher Screening, On the Job Evaluations and Performancee," DoQSS Working Papers 18-06, Quantitative Social Science - UCL Social Research Institute, University College London.
    7. Marianne Bitler & Sean Corcoran & Thurston Domina & Emily Penner, 2019. "Teacher Effects on Student Achievement and Height: A Cautionary Tale," NBER Working Papers 26480, National Bureau of Economic Research, Inc.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Canales, Andrea & Maldonado, Luis, 2018. "Teacher quality and student achievement in Chile: Linking teachers' contribution and observable characteristics," International Journal of Educational Development, Elsevier, vol. 60(C), pages 33-50.
    2. Naven, Matthew, 2019. "Human-Capital Formation During Childhood and Adolescence: Evidence from School Quality and Postsecondary Success in California," MPRA Paper 97716, University Library of Munich, Germany.
    3. Araujo P., Maria Daniela & Quis, Johanna Sophie, 2021. "Parents can tell! Evidence on classroom quality differences in German primary schools," BERG Working Paper Series 172, Bamberg University, Bamberg Economic Research Group.
    4. Araujo P., María Daniela & Quis, Johanna Sophie, 2021. "Teacher Effects in Germany: Evidence from Elementary School," VfS Annual Conference 2021 (Virtual Conference): Climate Economics 242457, Verein für Socialpolitik / German Economic Association.
    5. Hermann, Zoltán & Horváth, Hedvig, 2022. "Tanári eredményesség és tanár-diák összepárosítás az általános iskolákban. Empirikus mintázatok három magyarországi tankerület adatai alapján [Teacher effectiveness and teacher-student matching in ," Közgazdasági Szemle (Economic Review - monthly of the Hungarian Academy of Sciences), Közgazdasági Szemle Alapítvány (Economic Review Foundation), vol. 0(11), pages 1377-1406.
    6. Marc Steeg & Sander Gerritsen, 2016. "Teacher Evaluations and Pupil Achievement Gains: Evidence from Classroom Observations," De Economist, Springer, vol. 164(4), pages 419-443, December.
    7. Bacher-Hicks, Andrew & Chin, Mark J. & Kane, Thomas J. & Staiger, Douglas O., 2019. "An experimental evaluation of three teacher quality measures: Value-added, classroom observations, and student surveys," Economics of Education Review, Elsevier, vol. 73(C).
    8. Lee Crawfurd & Caine Rolleston, 2020. "Long‐run effects of teachers in developing countries," Review of Development Economics, Wiley Blackwell, vol. 24(4), pages 1279-1299, November.
    9. Papay, John P. & Kraft, Matthew A., 2015. "Productivity returns to experience in the teacher labor market: Methodological challenges and new evidence on long-term career improvement," Journal of Public Economics, Elsevier, vol. 130(C), pages 105-119.
    10. Goel, Deepti & Barooah, Bidisha, 2018. "Drivers of Student Performance: Evidence from Higher Secondary Public Schools in Delhi," GLO Discussion Paper Series 231, Global Labor Organization (GLO).
    11. Ingo E. Isphording & Ulf Zölitz, 2020. "The value of a peer," ECON - Working Papers 342, Department of Economics - University of Zurich.
    12. Filmer,Deon P. & Nahata,Vatsal & Sabarwal,Shwetlena, 2021. "Preparation, Practice, and Beliefs : A Machine Learning Approach to Understanding Teacher Effectiveness," Policy Research Working Paper Series 9847, The World Bank.
    13. Hanushek, Eric A. & Rivkin, Steven G. & Schiman, Jeffrey C., 2016. "Dynamic effects of teacher turnover on the quality of instruction," Economics of Education Review, Elsevier, vol. 55(C), pages 132-148.
    14. Christian Posso & Jorge Tamayo & Arlen Guarin & Estefania Saravia, 2024. "Luck of the Draw: The Causal Effect of Physicians on Birth Outcomes," Borradores de Economia 1269, Banco de la Republica de Colombia.
    15. David Blazar, 2018. "Validating Teacher Effects on Students’ Attitudes and Behaviors: Evidence from Random Assignment of Teachers to Students," Education Finance and Policy, MIT Press, vol. 13(3), pages 281-309, Summer.
    16. Tanaka, Ryuichi & Bessho, Shun-ichiro & Kawamura, Akira & Noguchi, Haruko & Ushijima, Koichi, 2020. "Determinants of Teacher Value-Added in Public Primary Schools: Evidence from Administrative Panel Data," IZA Discussion Papers 13146, Institute of Labor Economics (IZA).
    17. Zheng, Lei & Qi, Xiang & Zhang, Chongjiu, 2023. "Can improvements in teacher quality reduce the cognitive gap between urban and rural students in China?," International Journal of Educational Development, Elsevier, vol. 100(C).
    18. Evan Riehl & Meredith Welch, 2023. "Accountability, Test Prep Incentives, and the Design of Math and English Exams," Journal of Policy Analysis and Management, John Wiley & Sons, Ltd., vol. 42(1), pages 60-96, January.
    19. Bruns, Barbara & Costa, Leandro & Cunha, Nina, 2018. "Through the looking glass: Can classroom observation and coaching improve teacher performance in Brazil?," Economics of Education Review, Elsevier, vol. 64(C), pages 214-250.
    20. Jacob, Brian A. & Rockoff, Jonah E. & Taylor, Eric S. & Lindy, Benjamin & Rosen, Rachel, 2018. "Teacher applicant hiring and teacher performance: Evidence from DC public schools," Journal of Public Economics, Elsevier, vol. 166(C), pages 81-97.

    More about this item

    JEL classification:

    • I21 - Health, Education, and Welfare - - Education - - - Analysis of Education
    • J24 - Labor and Demographic Economics - - Demand and Supply of Labor - - - Human Capital; Skills; Occupational Choice; Labor Productivity

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:nbr:nberwo:23478. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: the person in charge (email available below). General contact details of provider: https://edirc.repec.org/data/nberrus.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.