IDEAS home Printed from https://ideas.repec.org/p/use/tkiwps/1416.html
   My bibliography  Save this paper

Measuring Teaching Quality in Higher Education: Assessing the Problem of Selection Bias in Course Evaluations

Author

Listed:
  • Anna Salomons
  • Maarten Goos

Abstract

Student evaluations of teaching are widely used to measure teaching quality and compare it across different courses, teachers, departments and institutions: as such, they are of increasing importance for teacher promotion decisions as well as student course selection. However, the response on course evaluations is rarely perfect, rendering such uses unwarranted if students who participate in the evaluation are not randomly selected: this paper is the first to investigate this issue. We quantify the direction and size of selection on both observable and unobservable characteristics for a large European university where course evaluation response rates differ across courses. Our results suggest course evaluations are upward biased, and that this bias mostly derives from selection on characteristics unlikely to be observed by the typical university administrator. Correcting for selection bias has sizable effects on both scores in any given course and the evaluation-based ranking of different courses.

Suggested Citation

  • Anna Salomons & Maarten Goos, 2014. "Measuring Teaching Quality in Higher Education: Assessing the Problem of Selection Bias in Course Evaluations," Working Papers 14-16, Utrecht School of Economics.
    • Handle: RePEc:use:tkiwps:1416
    as

    Download full text from publisher

    File URL: https://dspace.library.uu.nl/bitstream/handle/1874/306265/16_2.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Braga, Michela & Paccagnella, Marco & Pellizzari, Michele, 2014. "Evaluating students’ evaluations of professors," Economics of Education Review, Elsevier, vol. 41(C), pages 71-88.
    2. Giuseppe De Luca & Valeria Perotti, 2011. "Estimation of ordered response models with sample selection," Stata Journal, StataCorp LP, vol. 11(2), pages 213-239, June.
    3. Michael A. McPherson, 2006. "Determinants of How Students Evaluate Teachers," The Journal of Economic Education, Taylor & Francis Journals, vol. 37(1), pages 3-20, January.
    4. Heckman, James J, 1978. "Dummy Endogenous Variables in a Simultaneous Equation System," Econometrica, Econometric Society, vol. 46(4), pages 931-959, July.
    5. Florian Hoffmann & Philip Oreopoulos, 2009. "Professor Qualities and Student Achievement," The Review of Economics and Statistics, MIT Press, vol. 91(1), pages 83-92, February.
    6. Krautmann, Anthony C. & Sander, William, 1999. "Grades and student evaluations of teachers," Economics of Education Review, Elsevier, vol. 18(1), pages 59-63, February.
    7. Rosemary J. Avery & W. Keith Bryant & Alan Mathios & Hyojin Kang & Duncan Bell, 2006. "Electronic Course Evaluations: Does an Online Delivery System Influence Student Evaluations?," The Journal of Economic Education, Taylor & Francis Journals, vol. 37(1), pages 21-37, January.
    8. Scott E. Carrell & James E. West, 2010. "Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors," Journal of Political Economy, University of Chicago Press, vol. 118(3), pages 409-432, June.
    9. Imberman, Scott A. & Lovenheim, Michael F., 2016. "Does the market value value-added? Evidence from housing prices after a public release of school and teacher value-added," Journal of Urban Economics, Elsevier, vol. 91(C), pages 104-121.
    10. Sartori, Anne E., 2003. "An Estimator for Some Binary-Outcome Selection Models Without Exclusion Restrictions," Political Analysis, Cambridge University Press, vol. 11(2), pages 111-138, April.
    11. Alfonso Miranda & Sophia Rabe-Hesketh, 2006. "Maximum likelihood estimation of endogenous switching and sample selection models for binary, ordinal, and count variables," Stata Journal, StataCorp LP, vol. 6(3), pages 285-308, September.
    12. Van de Ven, Wynand P. M. M. & Van Praag, Bernard M. S., 1981. "The demand for deductibles in private health insurance : A probit model with sample selection," Journal of Econometrics, Elsevier, vol. 17(2), pages 229-252, November.
    13. William E. Becker & William Bosshardt & Michael Watts, 2012. "How Departments of Economics Evaluate Teaching," The Journal of Economic Education, Taylor & Francis Journals, vol. 43(3), pages 325-333, July.
    14. Beleche, Trinidad & Fairris, David & Marks, Mindy, 2012. "Do course evaluations truly reflect student learning? Evidence from an objectively graded post-test," Economics of Education Review, Elsevier, vol. 31(5), pages 709-719.
    15. Cunha, Jesse M. & Miller, Trey, 2014. "Measuring value-added in higher education: Possibilities and limitations in the use of administrative data," Economics of Education Review, Elsevier, vol. 42(C), pages 64-77.
    16. Michael A McPherson & R Todd Jewell & Myungsup Kim, 2009. "What Determines Student Evaluation Scores? A Random Effects Analysis of Undergraduate Economics Classes," Eastern Economic Journal, Palgrave Macmillan;Eastern Economic Association, vol. 35(1), pages 37-51.
    17. Heckman, James J, 1974. "Shadow Prices, Market Wages, and Labor Supply," Econometrica, Econometric Society, vol. 42(4), pages 679-694, July.
    18. Paul Isely & Harinder Singh, 2005. "Do Higher Grades Lead to Favorable Student Evaluations?," The Journal of Economic Education, Taylor & Francis Journals, vol. 36(1), pages 29-42, January.
    19. Ewing, Andrew M., 2012. "Estimating the impact of relative expected grade on student evaluations of teachers," Economics of Education Review, Elsevier, vol. 31(1), pages 141-154.
    20. Bruce A. Weinberg & Masanori Hashimoto & Belton M. Fleisher, 2009. "Evaluating Teaching in Higher Education," The Journal of Economic Education, Taylor & Francis Journals, vol. 40(3), pages 227-261, July.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Maarten Goos & Anna Salomons, 2017. "Measuring teaching quality in higher education: assessing selection bias in course evaluations," Research in Higher Education, Springer;Association for Institutional Research, vol. 58(4), pages 341-364, June.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Maarten Goos & Anna Salomons, 2017. "Measuring teaching quality in higher education: assessing selection bias in course evaluations," Research in Higher Education, Springer;Association for Institutional Research, vol. 58(4), pages 341-364, June.
    2. Rieger, Matthias & Voorvelt, Katherine, 2016. "Gender, ethnicity and teaching evaluations: Evidence from mixed teaching teamsAuthor-Name: Wagner, Natascha," Economics of Education Review, Elsevier, vol. 54(C), pages 79-94.
    3. Wagner, N. & Rieger, M. & Voorvelt, K.J., 2016. "Gender, ethnicity and teaching evaluations : Evidence from mixed teaching teams," ISS Working Papers - General Series 617, International Institute of Social Studies of Erasmus University Rotterdam (ISS), The Hague.
    4. Donghun Cho & Joonmo Cho, 2017. "Does More Accurate Knowledge of Course Grade Impact Teaching Evaluation?," Education Finance and Policy, MIT Press, vol. 12(2), pages 224-240, Spring.
    5. Gorry, Devon, 2017. "The impact of grade ceilings on student grades and course evaluations: Evidence from a policy change," Economics of Education Review, Elsevier, vol. 56(C), pages 133-140.
    6. Boring, Anne, 2017. "Gender biases in student evaluations of teaching," Journal of Public Economics, Elsevier, vol. 145(C), pages 27-41.
    7. Beleche, Trinidad & Fairris, David & Marks, Mindy, 2012. "Do course evaluations truly reflect student learning? Evidence from an objectively graded post-test," Economics of Education Review, Elsevier, vol. 31(5), pages 709-719.
    8. Anne Boring, 2015. "Gender Biases in student evaluations of teachers," Documents de Travail de l'OFCE 2015-13, Observatoire Francais des Conjonctures Economiques (OFCE).
    9. Ali Palali & Roel van Elk & Jonneke Bolhaar & Iryna Rud, 2017. "Are good researchers also good teachers? The relationship between research quality and teaching quality," CPB Discussion Paper 347, CPB Netherlands Bureau for Economic Policy Analysis.
    10. Ali Palali & Roel van Elk & Jonneke Bolhaar & Iryna Rud, 2017. "Are good researchers also good teachers? The relationship between research quality and teaching quality," CPB Discussion Paper 347.rdf, CPB Netherlands Bureau for Economic Policy Analysis.
    11. Braga, Michela & Paccagnella, Marco & Pellizzari, Michele, 2014. "Evaluating students’ evaluations of professors," Economics of Education Review, Elsevier, vol. 41(C), pages 71-88.
    12. repec:hal:spmain:info:hdl:2441/1seuirq4ak9b9bouu1j29ebui7 is not listed on IDEAS
    13. Palali, Ali & van Elk, Roel & Bolhaar, Jonneke & Rud, Iryna, 2018. "Are good researchers also good teachers? The relationship between research quality and teaching quality," Economics of Education Review, Elsevier, vol. 64(C), pages 40-49.
    14. Rodríguez, Rosa & Rubio, Gonzalo, 2016. "Teaching quality and academic research," International Review of Economics Education, Elsevier, vol. 23(C), pages 10-27.
    15. Angelo Antoci & Irene Brunetti & Pierluigi Sacco & Mauro Sodini, 2021. "Student evaluation of teaching, social influence dynamics, and teachers’ choices: An evolutionary model," Journal of Evolutionary Economics, Springer, vol. 31(1), pages 325-348, January.
    16. Aurora Garc𫑇allego & Nikolaos Georgantz & Joan Mart󻑍ontaner & Teodosio P鲥z-Amaral, 2015. "(How) Do research and administrative duties affect university professors' teaching?," Applied Economics, Taylor & Francis Journals, vol. 47(45), pages 4868-4883, September.
    17. James F. Ragan & Bhavneet Walia, 2010. "Differences in Student Evaluations of Principles and Other Economics Courses and the Allocation of Faculty across Courses," The Journal of Economic Education, Taylor & Francis Journals, vol. 41(4), pages 335-352, September.
    18. Cho, Donghun & Baek, Wonyoung & Cho, Joonmo, 2015. "Why do good performing students highly rate their instructors? Evidence from a natural experiment," Economics of Education Review, Elsevier, vol. 49(C), pages 172-179.
    19. Radchenko, Natalia, 2020. "Biases in Student Evaluations of Teaching: An American Case Study," IZA Discussion Papers 13603, Institute of Labor Economics (IZA).
    20. Lütkenhöner, Laura, 2013. "Können sich Hochschuldozenten bessere studentische Lehrevaluationen "erkaufen"?," Discussion Papers of the Institute for Organisational Economics 7/2013, University of Münster, Institute for Organisational Economics.
    21. Cannon, Edmund & Cipriani, Giam Pietro, 2021. "Gender Differences in Student Evaluations of Teaching: Identification and Consequences," IZA Discussion Papers 14387, Institute of Labor Economics (IZA).

    More about this item

    Keywords

    Educational economics; Student evaluations of teaching; Education quality; Sample selection;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:use:tkiwps:1416. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Marina Muilwijk (email available below). General contact details of provider: https://edirc.repec.org/data/eiruunl.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.