IDEAS home Printed from https://ideas.repec.org/a/spr/reihed/v58y2017i4d10.1007_s11162-016-9429-8.html
   My bibliography  Save this article

Measuring teaching quality in higher education: assessing selection bias in course evaluations

Author

Listed:
  • Maarten Goos

    () (Utrecht University School of Economics)

  • Anna Salomons

    () (Utrecht University School of Economics)

Abstract

Student evaluations of teaching (SETs) are widely used to measure teaching quality in higher education and compare it across different courses, teachers, departments and institutions. Indeed, SETs are of increasing importance for teacher promotion decisions, student course selection, as well as for auditing practices demonstrating institutional performance. However, survey response is typically low, rendering these uses unwarranted if students who respond to the evaluation are not randomly selected along observed and unobserved dimensions. This paper is the first to fully quantify this problem by analyzing the direction and size of selection bias resulting from both observed and unobserved characteristics for over 3000 courses taught in a large European university. We find that course evaluations are upward biased, and that correcting for selection bias has non-negligible effects on the average evaluation score and on the evaluation-based ranking of courses. Moreover, this bias mostly derives from selection on unobserved characteristics, implying that correcting evaluation scores for observed factors such as student grades does not solve the problem. However, we find that adjusting for selection only has small impacts on the measured effects of observables on SETs, validating a large related literature which considers the observable determinants of evaluation scores without correcting for selection bias.

Suggested Citation

  • Maarten Goos & Anna Salomons, 2017. "Measuring teaching quality in higher education: assessing selection bias in course evaluations," Research in Higher Education, Springer;Association for Institutional Research, vol. 58(4), pages 341-364, June.
  • Handle: RePEc:spr:reihed:v:58:y:2017:i:4:d:10.1007_s11162-016-9429-8
    DOI: 10.1007/s11162-016-9429-8
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11162-016-9429-8
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Steven G. Rivkin & Eric A. Hanushek & John F. Kain, 2005. "Teachers, Schools, and Academic Achievement," Econometrica, Econometric Society, vol. 73(2), pages 417-458, March.
    2. Meng, Chun-Lo & Schmidt, Peter, 1985. "On the Cost of Partial Observability in the Bivariate Probit Model," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 26(1), pages 71-85, February.
    3. Charles F. Manski, 1989. "Anatomy of the Selection Problem," Journal of Human Resources, University of Wisconsin Press, vol. 24(3), pages 343-360.
    4. Pigini Claudia, 2015. "Bivariate Non-Normality in the Sample Selection Model," Journal of Econometric Methods, De Gruyter, vol. 4(1), pages 1-22, January.
    5. Jonah E. Rockoff, 2004. "The Impact of Individual Teachers on Student Achievement: Evidence from Panel Data," American Economic Review, American Economic Association, vol. 94(2), pages 247-252, May.
    6. Braga, Michela & Paccagnella, Marco & Pellizzari, Michele, 2014. "Evaluating students’ evaluations of professors," Economics of Education Review, Elsevier, vol. 41(C), pages 71-88.
    7. Michael A. McPherson & R. Todd Jewell, 2007. "Leveling the Playing Field: Should Student Evaluation Scores be Adjusted?," Social Science Quarterly, Southwestern Social Science Association, vol. 88(3), pages 868-881, September.
    8. Angrist, Joshua D. & Guryan, Jonathan, 2008. "Does teacher testing raise teacher quality? Evidence from state certification requirements," Economics of Education Review, Elsevier, vol. 27(5), pages 483-503, October.
    9. Devesh Kapur & Megan Crowley, 2008. "Beyond the ABCs: Higher Education and Developing Countries," Working Papers 139, Center for Global Development.
    10. Giuseppe De Luca & Valeria Perotti, 2011. "Estimation of ordered response models with sample selection," Stata Journal, StataCorp LP, vol. 11(2), pages 213-239, June.
    11. Raj Chetty & John N. Friedman & Jonah E. Rockoff, 2014. "Measuring the Impacts of Teachers II: Teacher Value-Added and Student Outcomes in Adulthood," American Economic Review, American Economic Association, vol. 104(9), pages 2633-2679, September.
    12. Van de Ven, Wynand P. M. M. & Van Praag, Bernard M. S., 1981. "The demand for deductibles in private health insurance : A probit model with sample selection," Journal of Econometrics, Elsevier, vol. 17(2), pages 229-252, November.
    13. William E. Becker & William Bosshardt & Michael Watts, 2012. "How Departments of Economics Evaluate Teaching," The Journal of Economic Education, Taylor & Francis Journals, vol. 43(3), pages 325-333, July.
    14. Samer Kherfi, 2011. "Whose Opinion Is It Anyway? Determinants of Participation in Student Evaluation of Teaching," The Journal of Economic Education, Taylor & Francis Journals, vol. 42(1), pages 19-30, January.
    15. van der Klaauw, Bas & Koning, Ruud H, 2003. "Testing the Normality Assumption in the Sample Selection Model with an Application to Travel Demand," Journal of Business & Economic Statistics, American Statistical Association, vol. 21(1), pages 31-42, January.
    16. Arabmazar, Abbas & Schmidt, Peter, 1982. "An Investigation of the Robustness of the Tobit Estimator to Non-Normality," Econometrica, Econometric Society, vol. 50(4), pages 1055-1063, July.
    17. Patrick Puhani, 2000. "The Heckman Correction for Sample Selection and Its Critique," Journal of Economic Surveys, Wiley Blackwell, vol. 14(1), pages 53-68, February.
    18. Michael A. McPherson, 2006. "Determinants of How Students Evaluate Teachers," The Journal of Economic Education, Taylor & Francis Journals, vol. 37(1), pages 3-20, January.
    19. Heckman, James, 2013. "Sample selection bias as a specification error," Applied Econometrics, Russian Presidential Academy of National Economy and Public Administration (RANEPA), vol. 31(3), pages 129-137.
    20. Keane, Michael P, 1992. "A Note on Identification in the Multinomial Probit Model," Journal of Business & Economic Statistics, American Statistical Association, vol. 10(2), pages 193-200, April.
    21. Beleche, Trinidad & Fairris, David & Marks, Mindy, 2012. "Do course evaluations truly reflect student learning? Evidence from an objectively graded post-test," Economics of Education Review, Elsevier, vol. 31(5), pages 709-719.
    22. Brian A. Jacob & Lars Lefgren, 2008. "Can Principals Identify Effective Teachers? Evidence on Subjective Performance Evaluation in Education," Journal of Labor Economics, University of Chicago Press, vol. 26(1), pages 101-136.
    23. Heckman, James J, 1978. "Dummy Endogenous Variables in a Simultaneous Equation System," Econometrica, Econometric Society, vol. 46(4), pages 931-959, July.
    24. repec:hrv:faseco:30749073 is not listed on IDEAS
    25. Cunha, Jesse M. & Miller, Trey, 2014. "Measuring value-added in higher education: Possibilities and limitations in the use of administrative data," Economics of Education Review, Elsevier, vol. 42(C), pages 64-77.
    26. Michael A McPherson & R Todd Jewell & Myungsup Kim, 2009. "What Determines Student Evaluation Scores? A Random Effects Analysis of Undergraduate Economics Classes," Eastern Economic Journal, Palgrave Macmillan;Eastern Economic Association, vol. 35(1), pages 37-51.
    27. Langbein, Laura, 2008. "Management by results: Student evaluation of faculty teaching and the mis-measurement of performance," Economics of Education Review, Elsevier, vol. 27(4), pages 417-428, August.
    28. Raj Chetty & John N. Friedman & Jonah E. Rockoff, 2014. "Measuring the Impacts of Teachers I: Evaluating Bias in Teacher Value-Added Estimates," American Economic Review, American Economic Association, vol. 104(9), pages 2593-2632, September.
    29. Florian Hoffmann & Philip Oreopoulos, 2009. "Professor Qualities and Student Achievement," The Review of Economics and Statistics, MIT Press, vol. 91(1), pages 83-92, February.
    30. Krautmann, Anthony C. & Sander, William, 1999. "Grades and student evaluations of teachers," Economics of Education Review, Elsevier, vol. 18(1), pages 59-63, February.
    31. Heckman, James J, 1974. "Shadow Prices, Market Wages, and Labor Supply," Econometrica, Econometric Society, vol. 42(4), pages 679-694, July.
    32. Kane, Thomas J. & Rockoff, Jonah E. & Staiger, Douglas O., 2008. "What does certification tell us about teacher effectiveness? Evidence from New York City," Economics of Education Review, Elsevier, vol. 27(6), pages 615-631, December.
    33. Francis Vella, 1998. "Estimating Models with Sample Selection Bias: A Survey," Journal of Human Resources, University of Wisconsin Press, vol. 33(1), pages 127-169.
    34. Paul Isely & Harinder Singh, 2005. "Do Higher Grades Lead to Favorable Student Evaluations?," The Journal of Economic Education, Taylor & Francis Journals, vol. 36(1), pages 29-42, January.
    35. Erik Hanushek & F. Welch (ed.), 2006. "Handbook of the Economics of Education," Handbook of the Economics of Education, Elsevier, edition 1, volume 1, number 1, June.
    36. Gronau, Reuben, 1974. "Wage Comparisons-A Selectivity Bias," Journal of Political Economy, University of Chicago Press, vol. 82(6), pages 1119-1143, Nov.-Dec..
    37. Erik Hanushek & F. Welch (ed.), 2006. "Handbook of the Economics of Education," Handbook of the Economics of Education, Elsevier, edition 1, volume 2, number 2, June.
    38. Rosemary J. Avery & W. Keith Bryant & Alan Mathios & Hyojin Kang & Duncan Bell, 2006. "Electronic Course Evaluations: Does an Online Delivery System Influence Student Evaluations?," The Journal of Economic Education, Taylor & Francis Journals, vol. 37(1), pages 21-37, January.
    39. Scott E. Carrell & James E. West, 2010. "Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors," Journal of Political Economy, University of Chicago Press, vol. 118(3), pages 409-432, June.
    40. repec:hrv:faseco:30749606 is not listed on IDEAS
    41. Ewing, Andrew M., 2012. "Estimating the impact of relative expected grade on student evaluations of teachers," Economics of Education Review, Elsevier, vol. 31(1), pages 141-154.
    42. Bruce A. Weinberg & Masanori Hashimoto & Belton M. Fleisher, 2009. "Evaluating Teaching in Higher Education," The Journal of Economic Education, Taylor & Francis Journals, vol. 40(3), pages 227-261, July.
    43. Anna Salomons & Maarten Goos, 2014. "Measuring Teaching Quality in Higher Education : Assessing the Problem of Selection Bias in Course Evaluations," Working Papers 14-16, Utrecht School of Economics.
    44. Berg, Nathan, 2005. "Non-response bias," MPRA Paper 26373, University Library of Munich, Germany.
    45. Alfonso Miranda & Sophia Rabe-Hesketh, 2006. "Maximum likelihood estimation of endogenous switching and sample selection models for binary, ordinal, and count variables," Stata Journal, StataCorp LP, vol. 6(3), pages 285-308, September.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Anna Salomons & Maarten Goos, 2014. "Measuring Teaching Quality in Higher Education : Assessing the Problem of Selection Bias in Course Evaluations," Working Papers 14-16, Utrecht School of Economics.
    2. Rieger, Matthias & Voorvelt, Katherine, 2016. "Gender, ethnicity and teaching evaluations: Evidence from mixed teaching teamsAuthor-Name: Wagner, Natascha," Economics of Education Review, Elsevier, vol. 54(C), pages 79-94.
    3. Donghun Cho & Joonmo Cho, 2017. "Does More Accurate Knowledge of Course Grade Impact Teaching Evaluation?," Education Finance and Policy, MIT Press, vol. 12(2), pages 224-240, Spring.
    4. Tian, Zhilei & Wei, Yi & Li, Fang, 2019. "Who are better teachers? The effects of tenure-track and part-time faculty on student achievement," China Economic Review, Elsevier, vol. 53(C), pages 140-151.
    5. Wagner, N. & Rieger, M. & Voorvelt, K.J., 2016. "Gender, ethnicity and teaching evaluations : Evidence from mixed teaching teams," ISS Working Papers - General Series 617, International Institute of Social Studies of Erasmus University Rotterdam (ISS), The Hague.
    6. Gorry, Devon, 2017. "The impact of grade ceilings on student grades and course evaluations: Evidence from a policy change," Economics of Education Review, Elsevier, vol. 56(C), pages 133-140.
    7. Braga, Michela & Paccagnella, Marco & Pellizzari, Michele, 2014. "Evaluating students’ evaluations of professors," Economics of Education Review, Elsevier, vol. 41(C), pages 71-88.
    8. Boring, Anne, 2017. "Gender biases in student evaluations of teaching," Journal of Public Economics, Elsevier, vol. 145(C), pages 27-41.
    9. Beleche, Trinidad & Fairris, David & Marks, Mindy, 2012. "Do course evaluations truly reflect student learning? Evidence from an objectively graded post-test," Economics of Education Review, Elsevier, vol. 31(5), pages 709-719.
    10. Balcázar, Carlos Felipe & Nopo, Hugo R., 2014. "Broken Gears: The Value Added of Higher Education on Teachers' Academic Achievement," IZA Discussion Papers 8477, Institute of Labor Economics (IZA).
    11. Eric A. Hanushek & Marc Piopiunik & Simon Wiederhold, 2019. "The Value of Smarter Teachers: International Evidence on Teacher Cognitive Skills and Student Performance," Journal of Human Resources, University of Wisconsin Press, vol. 54(4), pages 857-899.
    12. De Witte, Kristof & Rogge, Nicky, 2011. "Accounting for exogenous influences in performance evaluations of teachers," Economics of Education Review, Elsevier, vol. 30(4), pages 641-653, August.
    13. Rodríguez, Rosa & Rubio, Gonzalo, 2016. "Teaching quality and academic research," International Review of Economics Education, Elsevier, vol. 23(C), pages 10-27.
    14. Ali Palali & Roel van Elk & Jonneke Bolhaar & Iryna Rud, 2017. "Are good researchers also good teachers? The relationship between research quality and teaching quality," CPB Discussion Paper 347, CPB Netherlands Bureau for Economic Policy Analysis.
    15. Naven, Matthew, 2019. "Human-Capital Formation During Childhood and Adolescence: Evidence from School Quality and Postsecondary Success in California," MPRA Paper 97716, University Library of Munich, Germany.
    16. Cho, Donghun & Baek, Wonyoung & Cho, Joonmo, 2015. "Why do good performing students highly rate their instructors? Evidence from a natural experiment," Economics of Education Review, Elsevier, vol. 49(C), pages 172-179.
    17. Omar Paccagnella, 2011. "Anchoring vignettes with sample selection due to non‐response," Journal of the Royal Statistical Society Series A, Royal Statistical Society, vol. 174(3), pages 665-687, July.
    18. Eric A. Hanushek & Marc Piopiunik & Simon Wiederhold, 2014. "International Evidence on Teacher Cognitive Skills and Student Performance," CID Working Papers 63, Center for International Development at Harvard University.
    19. Palali, Ali & van Elk, Roel & Bolhaar, Jonneke & Rud, Iryna, 2018. "Are good researchers also good teachers? The relationship between research quality and teaching quality," Economics of Education Review, Elsevier, vol. 64(C), pages 40-49.
    20. Radchenko, Natalia, 2020. "Biases in Student Evaluations of Teaching: An American Case Study," IZA Discussion Papers 13603, Institute of Labor Economics (IZA).

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:reihed:v:58:y:2017:i:4:d:10.1007_s11162-016-9429-8. See general information about how to correct material in RePEc.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: (Sonal Shukla) or (Springer Nature Abstracting and Indexing). General contact details of provider: http://www.springer.com .

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service hosted by the Research Division of the Federal Reserve Bank of St. Louis . RePEc uses bibliographic data supplied by the respective publishers.