IDEAS home Printed from https://ideas.repec.org/a/spr/reihed/v58y2017i5d10.1007_s11162-016-9437-8.html
   My bibliography  Save this article

A Criterion-Referenced Approach to Student Ratings of Instruction

Author

Listed:
  • J. Patrick Meyer

    (University of Virginia)

  • Justin B. Doromal

    (University of Virginia)

  • Xiaoxin Wei

    (University of Virginia)

  • Shi Zhu

    (University of Virginia)

Abstract

We developed a criterion-referenced student rating of instruction (SRI) to facilitate formative assessment of teaching. It involves four dimensions of teaching quality that are grounded in current instructional design principles: Organization and structure, Assessment and feedback, Personal interactions, and Academic rigor. Using item response theory and Wright mapping methods, we describe teaching characteristics at various points along the latent continuum for each scale. These maps enable criterion-referenced score interpretation by making an explicit connection between test performance and the theoretical framework. We explain the way our Wright maps can be used to enhance an instructor’s ability to interpret scores and identify ways to refine teaching. Although our work is aimed at improving score interpretation, a criterion-referenced test is not immune to factors that may bias test scores. The literature on SRIs is filled with research on factors unrelated to teaching that may bias scores. Therefore, we also used multilevel models to evaluate the extent to which student and course characteristic may affect scores and compromise score interpretation. Results indicated that student anger and the interaction between student gender and instructor gender are significant effects that account for a small amount of variance in SRI scores. All things considered, our criterion-referenced approach to SRIs is a viable way to describe teaching quality and help instructors refine pedagogy and facilitate course development.

Suggested Citation

  • J. Patrick Meyer & Justin B. Doromal & Xiaoxin Wei & Shi Zhu, 2017. "A Criterion-Referenced Approach to Student Ratings of Instruction," Research in Higher Education, Springer;Association for Institutional Research, vol. 58(5), pages 545-567, August.
    • Handle: RePEc:spr:reihed:v:58:y:2017:i:5:d:10.1007_s11162-016-9437-8
    DOI: 10.1007/s11162-016-9437-8
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11162-016-9437-8
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.
    File URL: https://libkey.io/10.1007/s11162-016-9437-8?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Braga, Michela & Paccagnella, Marco & Pellizzari, Michele, 2014. "Evaluating students’ evaluations of professors," Economics of Education Review, Elsevier, vol. 41(C), pages 71-88.
    2. Geoff Masters, 1982. "A rasch model for partial credit scoring," Psychometrika, Springer;The Psychometric Society, vol. 47(2), pages 149-174, June.
    3. Scott E. Carrell & James E. West, 2010. "Does Professor Quality Matter? Evidence from Random Assignment of Students to Professors," Journal of Political Economy, University of Chicago Press, vol. 118(3), pages 409-432, June.
    4. Huynh Huynh, 1998. "On Score Locations of Binary and Partial Credit Items and Their Applications to Item Mapping and Criterion-Referenced Interpretation," Journal of Educational and Behavioral Statistics, , vol. 23(1), pages 35-56, March.
    5. Bates, Douglas & Mächler, Martin & Bolker, Ben & Walker, Steve, 2015. "Fitting Linear Mixed-Effects Models Using lme4," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 67(i01).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Beleche, Trinidad & Fairris, David & Marks, Mindy, 2012. "Do course evaluations truly reflect student learning? Evidence from an objectively graded post-test," Economics of Education Review, Elsevier, vol. 31(5), pages 709-719.
    2. Scott E. Carrell & Michal Kurlaender, 2023. "My Professor Cares: Experimental Evidence on the Role of Faculty Engagement," American Economic Journal: Economic Policy, American Economic Association, vol. 15(4), pages 113-141, November.
    3. Friederike Mengel & Jan Sauermann & Ulf Zölitz, 2019. "Gender Bias in Teaching Evaluations," Journal of the European Economic Association, European Economic Association, vol. 17(2), pages 535-566.
    4. Marta De Philippis, 2021. "Multi-Task Agents and Incentives: The Case of Teaching and Research for University Professors," The Economic Journal, Royal Economic Society, vol. 131(636), pages 1643-1681.
    5. Michela Braga & Marco Paccagnella & Michele Pellizzari, 2016. "The Impact of College Teaching on Students’ Academic and Labor Market Outcomes," Journal of Labor Economics, University of Chicago Press, vol. 34(3), pages 781-822.
    6. Abílio António Freitas Belo, 2017. "The evaluation of the Higher Education lecturer competences in East Timor: a student inquiry," Review of Social Sciences, LAR Center Press, vol. 2(1), pages 36-51, January.
    7. Michela Braga & Marco Paccagnella & Michele Pellizzari, 2014. "The academic and labor market returns of university professors," Temi di discussione (Economic working papers) 981, Bank of Italy, Economic Research and International Relations Area.
    8. Rodríguez, Rosa & Rubio, Gonzalo, 2016. "Teaching quality and academic research," International Review of Economics Education, Elsevier, vol. 23(C), pages 10-27.
    9. Ali Palali & Roel van Elk & Jonneke Bolhaar & Iryna Rud, 2017. "Are good researchers also good teachers? The relationship between research quality and teaching quality," CPB Discussion Paper 347, CPB Netherlands Bureau for Economic Policy Analysis.
    10. Rieger, Matthias & Voorvelt, Katherine, 2016. "Gender, ethnicity and teaching evaluations: Evidence from mixed teaching teamsAuthor-Name: Wagner, Natascha," Economics of Education Review, Elsevier, vol. 54(C), pages 79-94.
    11. Angelo Antoci & Irene Brunetti & Pierluigi Sacco & Mauro Sodini, 2021. "Student evaluation of teaching, social influence dynamics, and teachers’ choices: An evolutionary model," Journal of Evolutionary Economics, Springer, vol. 31(1), pages 325-348, January.
    12. Anna Salomons & Maarten Goos, 2014. "Measuring Teaching Quality in Higher Education: Assessing the Problem of Selection Bias in Course Evaluations," Working Papers 14-16, Utrecht School of Economics.
    13. Keng, Shao-Hsun, 2020. "Gender bias and statistical discrimination against female instructors in student evaluations of teaching," Labour Economics, Elsevier, vol. 66(C).
    14. Buurman, Margaretha & Delfgaauw, Josse & Dur, Robert & Zoutenbier, Robin, 2020. "When do teachers respond to student feedback? Evidence from a field experiment," Labour Economics, Elsevier, vol. 65(C).
    15. Ali Palali & Roel van Elk & Jonneke Bolhaar & Iryna Rud, 2017. "Are good researchers also good teachers? The relationship between research quality and teaching quality," CPB Discussion Paper 347.rdf, CPB Netherlands Bureau for Economic Policy Analysis.
    16. Insler, Michael & McQuoid, Alexander F. & Rahman, Ahmed S. & Smith, Katherine, 2021. "Fear and Loathing in the Classroom: Why Does Teacher Quality Matter?," IZA Discussion Papers 14036, Institute of Labor Economics (IZA).
    17. Falch, Torberg & Iversen, Jon Marius Vaag & Nyhus, Ole Henning & Strøm, Bjarne, 2022. "Quality measures in higher education: Norwegian evidence," Economics of Education Review, Elsevier, vol. 87(C).
    18. Gorry, Devon, 2017. "The impact of grade ceilings on student grades and course evaluations: Evidence from a policy change," Economics of Education Review, Elsevier, vol. 56(C), pages 133-140.
    19. Shao-Hsun Keng, 2018. "Tenure system and its impact on grading leniency, teaching effectiveness and student effort," Empirical Economics, Springer, vol. 55(3), pages 1207-1227, November.
    20. Maarten Goos & Anna Salomons, 2017. "Measuring teaching quality in higher education: assessing selection bias in course evaluations," Research in Higher Education, Springer;Association for Institutional Research, vol. 58(4), pages 341-364, June.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:reihed:v:58:y:2017:i:5:d:10.1007_s11162-016-9437-8. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.