IDEAS home Printed from https://ideas.repec.org/a/spr/psycho/v86y2021i2d10.1007_s11336-021-09759-0.html
   My bibliography  Save this article

Certainty-Based Marking on Multiple-Choice Items: Psychometrics Meets Decision Theory

Author

Listed:
  • Qian Wu

    (KU Leuven)

  • Monique Vanerum

    (University of Hasselt)

  • Anouk Agten

    (University of Hasselt)

  • Andrés Christiansen

    (KU Leuven)

  • Frank Vandenabeele

    (University of Hasselt)

  • Jean-Michel Rigo

    (University of Hasselt)

  • Rianne Janssen

    (KU Leuven)

Abstract

When a response to a multiple-choice item consists of selecting a single-best answer, it is not possible for examiners to differentiate between a response that is a product of knowledge and one that is largely a product of uncertainty. Certainty-based marking (CBM) is one testing format that requires examinees to express their degree of certainty on the response option they have selected, leading to an item score that depends both on the correctness of an answer and the certainty expressed. The expected score is maximized if examinees truthfully report their level of certainty. However, prospect theory states that people do not always make rational choices of the optimal outcome due to varying risk attitudes. By integrating a psychometric model and a decision-making perspective, the present study looks into the response behaviors of 334 first-year students of physiotherapy on six multiple-choice examinations with CBM in a case study. We used item response theory to model the objective probability of students giving a correct response to an item, and cumulative prospect theory to estimate their risk attitudes when students choose to report their certainty. The results showed that with the given CBM scoring matrix, students’ choices of a certainty level were affected by their risk attitudes. Students were generally risk averse and loss averse when they had a high success probability on an item, leading to an under-reporting of their certainty. Meanwhile, they were risk seeking in case of small success probabilities on the items, resulting in the over-reporting of certainty.

Suggested Citation

  • Qian Wu & Monique Vanerum & Anouk Agten & Andrés Christiansen & Frank Vandenabeele & Jean-Michel Rigo & Rianne Janssen, 2021. "Certainty-Based Marking on Multiple-Choice Items: Psychometrics Meets Decision Theory," Psychometrika, Springer;The Psychometric Society, vol. 86(2), pages 518-543, June.
  • Handle: RePEc:spr:psycho:v:86:y:2021:i:2:d:10.1007_s11336-021-09759-0
    DOI: 10.1007/s11336-021-09759-0
    as

    Download full text from publisher

    File URL: http://link.springer.com/10.1007/s11336-021-09759-0
    File Function: Abstract
    Download Restriction: Access to the full text of the articles in this series is restricted.

    File URL: https://libkey.io/10.1007/s11336-021-09759-0?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to search for a different version of it.

    References listed on IDEAS

    as
    1. Ryan O. Murphy & Robert H.W. ten Brincke, "undated". "Hierarchical maximum likelihood parameter estimation for cumulative prospect theory: Improving the reliability of individual risk parameter estimates," Working Papers ETH-RC-14-005, ETH Zurich, Chair of Systems Design.
    2. Adam Booij & Bernard Praag & Gijs Kuilen, 2010. "A parametric analysis of prospect theory’s functionals for the general population," Theory and Decision, Springer, vol. 68(1), pages 115-148, February.
    3. Tversky, Amos & Kahneman, Daniel, 1992. "Advances in Prospect Theory: Cumulative Representation of Uncertainty," Journal of Risk and Uncertainty, Springer, vol. 5(4), pages 297-323, October.
    4. Anna Brown, 2016. "Item Response Models for Forced-Choice Questionnaires: A Common Framework," Psychometrika, Springer;The Psychometric Society, vol. 81(1), pages 135-160, March.
    5. Thomas Dohmen & Armin Falk & David Huffman & Uwe Sunde & Jürgen Schupp & Gert G. Wagner, 2011. "Individual Risk Attitudes: Measurement, Determinants, And Behavioral Consequences," Journal of the European Economic Association, European Economic Association, vol. 9(3), pages 522-550, June.
    6. McFadden, Daniel, 1980. "Econometric Models for Probabilistic Choice among Products," The Journal of Business, University of Chicago Press, vol. 53(3), pages 13-29, July.
    7. Sturtz, Sibylle & Ligges, Uwe & Gelman, Andrew, 2005. "R2WinBUGS: A Package for Running WinBUGS from R," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 12(i03).
    8. Train,Kenneth E., 2009. "Discrete Choice Methods with Simulation," Cambridge Books, Cambridge University Press, number 9780521766555.
    9. Daniel Kahneman & Amos Tversky, 2013. "Prospect Theory: An Analysis of Decision Under Risk," World Scientific Book Chapters, in: Leonard C MacLean & William T Ziemba (ed.), HANDBOOK OF THE FUNDAMENTALS OF FINANCIAL DECISION MAKING Part I, chapter 6, pages 99-127, World Scientific Publishing Co. Pte. Ltd..
    10. Steven Andrew Culpepper & James Joseph Balamuta, 2017. "A Hierarchical Model for Accuracy and Choice on Standardized Tests," Psychometrika, Springer;The Psychometric Society, vol. 82(3), pages 820-845, September.
    11. Chalmers, R. Philip, 2012. "mirt: A Multidimensional Item Response Theory Package for the R Environment," Journal of Statistical Software, Foundation for Open Access Statistics, vol. 48(i06).
    12. Shlomo Benartzi & Richard H. Thaler, 1995. "Myopic Loss Aversion and the Equity Premium Puzzle," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 110(1), pages 73-92.
    13. Rachel Croson & Uri Gneezy, 2009. "Gender Differences in Preferences," Journal of Economic Literature, American Economic Association, vol. 47(2), pages 448-474, June.
    14. Jeffrey Rouder & Jordan Province & Richard Morey & Pablo Gomez & Andrew Heathcote, 2015. "The Lognormal Race: A Cognitive-Process Model of Choice and Latency with Desirable Psychometric Properties," Psychometrika, Springer;The Psychometric Society, vol. 80(2), pages 491-513, June.
    15. Pekkarinen, Tuomas, 2015. "Gender differences in behaviour under competitive pressure: Evidence on omission patterns in university entrance examinations," Journal of Economic Behavior & Organization, Elsevier, vol. 115(C), pages 94-110.
    16. Henry Stott, 2006. "Cumulative prospect theory's functional menagerie," Journal of Risk and Uncertainty, Springer, vol. 32(2), pages 101-130, March.
    17. Ryan O. Murphy & Robert H. W. ten Brincke, 2018. "Hierarchical Maximum Likelihood Parameter Estimation for Cumulative Prospect Theory: Improving the Reliability of Individual Risk Parameter Estimates," Management Science, INFORMS, vol. 64(1), pages 308-328, January.
    18. Drazen Prelec, 1998. "The Probability Weighting Function," Econometrica, Econometric Society, vol. 66(3), pages 497-528, May.
    19. Maya Bar-Hillel & David Budescu & Yigal Attali, 2005. "Scoring and keying multiple choice tests: A case study in irrationality," Mind & Society: Cognitive Studies in Economics and Social Sciences, Springer;Fondazione Rosselli, vol. 4(1), pages 3-12, June.
    20. Kobberling, Veronika & Wakker, Peter P., 2005. "An index of loss aversion," Journal of Economic Theory, Elsevier, vol. 122(1), pages 119-131, May.
    21. Justin Sydnor, 2010. "(Over)insuring Modest Risks," American Economic Journal: Applied Economics, American Economic Association, vol. 2(4), pages 177-199, October.
    22. David Budescu & Yuanchao Bo, 2015. "Analyzing Test-Taking Behavior: Decision Theory Meets Psychometric Theory," Psychometrika, Springer;The Psychometric Society, vol. 80(4), pages 1105-1122, December.
    23. Katherine Baldiga, 2014. "Gender Differences in Willingness to Guess," Management Science, INFORMS, vol. 60(2), pages 434-448, February.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Špela Smrkolj & Enja Bančov & Vladimir Smrkolj, 2022. "The Reliability and Medical Students’ Appreciation of Certainty-Based Marking," IJERPH, MDPI, vol. 19(3), pages 1-11, February.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Aurélien Baillon & Han Bleichrodt & Vitalie Spinu, 2020. "Searching for the Reference Point," Management Science, INFORMS, vol. 66(1), pages 93-112, January.
    2. Ferdinand M. Vieider & Peter Martinsson & Pham Khanh Nam & Nghi Truong, 2019. "Risk preferences and development revisited," Theory and Decision, Springer, vol. 86(1), pages 1-21, February.
    3. Filiz-Ozbay, Emel & Guryan, Jonathan & Hyndman, Kyle & Kearney, Melissa & Ozbay, Erkut Y., 2015. "Do lottery payments induce savings behavior? Evidence from the lab," Journal of Public Economics, Elsevier, vol. 126(C), pages 1-24.
    4. Jakusch, Sven Thorsten & Meyer, Steffen & Hackethal, Andreas, 2019. "Taming models of prospect theory in the wild? Estimation of Vlcek and Hens (2011)," SAFE Working Paper Series 146, Leibniz Institute for Financial Research SAFE, revised 2019.
    5. Kpegli, Yao Thibaut & Corgnet, Brice & Zylbersztejn, Adam, 2023. "All at once! A comprehensive and tractable semi-parametric method to elicit prospect theory components," Journal of Mathematical Economics, Elsevier, vol. 104(C).
    6. Ranoua Bouchouicha & Lachlan Deer & Ashraf Galal Eid & Peter McGee & Daniel Schoch & Hrvoje Stojic & Jolanda Ygosse-Battisti & Ferdinand M. Vieider, 2019. "Gender effects for loss aversion: Yes, no, maybe?," Journal of Risk and Uncertainty, Springer, vol. 59(2), pages 171-184, October.
    7. Ryan O. Murphy & Robert H. W. ten Brincke, 2018. "Hierarchical Maximum Likelihood Parameter Estimation for Cumulative Prospect Theory: Improving the Reliability of Individual Risk Parameter Estimates," Management Science, INFORMS, vol. 64(1), pages 308-328, January.
    8. Simon Gächter & Eric J. Johnson & Andreas Herrmann, 2022. "Individual-level loss aversion in riskless and risky choices," Theory and Decision, Springer, vol. 92(3), pages 599-624, April.
    9. Hans-Martin von Gaudecker & Arthur van Soest & Erik Wengstrom, 2011. "Heterogeneity in Risky Choice Behavior in a Broad Population," American Economic Review, American Economic Association, vol. 101(2), pages 664-694, April.
    10. Adam Booij & Bernard Praag & Gijs Kuilen, 2010. "A parametric analysis of prospect theory’s functionals for the general population," Theory and Decision, Springer, vol. 68(1), pages 115-148, February.
    11. Lasha Lanchava & Kyle Carlson & Blanka Šebánková & Jaroslav Flegr & Gideon Nave, 2015. "No Evidence of Association between Toxoplasma gondii Infection and Financial Risk Taking in Females," PLOS ONE, Public Library of Science, vol. 10(9), pages 1-17, September.
    12. Laurent Denant-Boemont & Olivier L’Haridon, 2013. "La rationalité à l'épreuve de l'économie comportementale," Revue française d'économie, Presses de Sciences-Po, vol. 0(2), pages 35-89.
    13. Jakusch, Sven Thorsten, 2017. "On the applicability of maximum likelihood methods: From experimental to financial data," SAFE Working Paper Series 148, Leibniz Institute for Financial Research SAFE, revised 2017.
    14. Alam, Jessica & Georgalos, Konstantinos & Rolls, Harrison, 2022. "Risk preferences, gender effects and Bayesian econometrics," Journal of Economic Behavior & Organization, Elsevier, vol. 202(C), pages 168-183.
    15. Simon Gaechter & Eric Johnson & Andreas Herrmann, 2007. "Individual-Level Loss Aversion In Riskless And Risky Choices," Discussion Papers 2007-02, The Centre for Decision Research and Experimental Economics, School of Economics, University of Nottingham.
    16. Víctor González-Jiménez, 2021. "Incentive contracts when agents distort probabilities," Vienna Economics Papers vie2101, University of Vienna, Department of Economics.
    17. Syngjoo Choi & Jeongbin Kim & Eungik Lee & Jungmin Lee, 2022. "Probability Weighting and Cognitive Ability," Management Science, INFORMS, vol. 68(7), pages 5201-5215, July.
    18. Andersen, Steffen & Harrison, Glenn W. & Lau, Morten Igel & Rutström, Elisabet E., 2010. "Behavioral econometrics for psychologists," Journal of Economic Psychology, Elsevier, vol. 31(4), pages 553-576, August.
    19. Li, Baibing & Hensher, David A., 2017. "Risky weighting in discrete choice," Transportation Research Part B: Methodological, Elsevier, vol. 102(C), pages 1-21.
    20. Aurélien Baillon & Han Bleichrodt & Umut Keskin & Olivier l’Haridon & Chen Li, 2018. "The Effect of Learning on Ambiguity Attitudes," Management Science, INFORMS, vol. 64(5), pages 2181-2198, May.

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:spr:psycho:v:86:y:2021:i:2:d:10.1007_s11336-021-09759-0. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Sonal Shukla or Springer Nature Abstracting and Indexing (email available below). General contact details of provider: http://www.springer.com .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.