IDEAS home Printed from https://ideas.repec.org/a/eee/jeborg/v237y2025ics0167268125003038.html

Hedging-based scoring rules for multiple-choice questions

Author

Listed:
  • Fu, Jingcheng
  • Zhang, Xing
  • Zhong, Songfa

Abstract

This paper proposes two novel scoring rules for multiple-choice questions based on the test-takers’ propensity to hedge across possible answers. To examine these scoring rules, we randomly assign 2,986 participants in an IQ test into three conditions. In the control condition, participants choose one option, and receive one point for a correct response. In the treatment conditions, they can explicitly hedge by choosing k options: if the correct option is among the k chosen options, they receive 1/k point in the outcome-mixing treatment, and one point with probability 1/k in the probability-mixing treatment. We find that participants in both treatments hedge pervasively, score lower compared to those in the control, but do not improve their time allocation. While scores in the three conditions exhibit similar psychometric quality, we observe a significant correlation between academic performance and IQ score measured in the probability-mixing condition. These results highlight the potential of these two hedging-based scoring rules for practical applications in multiple-choice questions.

Suggested Citation

  • Fu, Jingcheng & Zhang, Xing & Zhong, Songfa, 2025. "Hedging-based scoring rules for multiple-choice questions," Journal of Economic Behavior & Organization, Elsevier, vol. 237(C).
    • Handle: RePEc:eee:jeborg:v:237:y:2025:i:c:s0167268125003038
    DOI: 10.1016/j.jebo.2025.107184
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S0167268125003038
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.jebo.2025.107184?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to

    for a different version of it.

    References listed on IDEAS

    as
    1. Aurélien Baillon & Yoram Halevy & Chen Li, 2022. "Experimental elicitation of ambiguity attitude using the random incentive system," Experimental Economics, Springer;Economic Science Association, vol. 25(3), pages 1002-1023, June.
    2. Lex Borghans & Bart Golsteyn & James J. Heckman & John Eric Humphries, 2016. "What Grades and Achievement Tests Measure," Working Papers 2016-022, Human Capital and Economic Opportunity Working Group.
    3. Konovalov, Arkady & Krajbich, Ian, 2019. "Revealed strength of preference: Inference from response times," Judgment and Decision Making, Cambridge University Press, vol. 14(4), pages 381-394, July.
    4. Uri Gneezy & Jan Potters, 1997. "An Experiment on Risk Taking and Evaluation Periods," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 112(2), pages 631-645.
    5. Thomas Dohmen & Armin Falk & David Huffman & Uwe Sunde, 2010. "Are Risk Aversion and Impatience Related to Cognitive Ability?," American Economic Review, American Economic Association, vol. 100(3), pages 1238-1260, June.
    6. Richard H. Thaler & Amos Tversky & Daniel Kahneman & Alan Schwartz, 1997. "The Effect of Myopia and Loss Aversion on Risk Taking: An Experimental Test," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 112(2), pages 647-661.
    7. Heiko Karle & Dirk Engelmann & Martin Peitz, 2022. "Student performance and loss aversion," Scandinavian Journal of Economics, Wiley Blackwell, vol. 124(2), pages 420-456, April.
    8. Dalia L. Diab & Michael A. Gillespie & Scott Highhouse, 2008. "Are maximizers really unhappy? The measurement of maximizing tendency," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 3, pages 364-370, June.
    9. Ola Andersson & Håkan J. Holm & Jean-Robert Tyran & Erik Wengström, 2016. "Risk Aversion Relates to Cognitive Ability: Preferences Or Noise?," Journal of the European Economic Association, European Economic Association, vol. 14(5), pages 1129-1154.
    10. Aurélien Baillon & Yoram Halevy & Chen Li, 2022. "Randomize at Your Own Risk: On the Observability of Ambiguity Aversion," Econometrica, Econometric Society, vol. 90(3), pages 1085-1107, May.
    11. Lilleholt, Lau, 2019. "Cognitive ability and risk aversion: A systematic review and meta analysis," Judgment and Decision Making, Cambridge University Press, vol. 14(3), pages 234-279, May.
    12. Thomas Dohmen & Armin Falk & David Huffman & Uwe Sunde, 2018. "On the Relationship between Cognitive Ability and Risk Preference," Journal of Economic Perspectives, American Economic Association, vol. 32(2), pages 115-134, Spring.
    13. Rubinstein, Ariel, 2002. "Irrational diversification in multiple decision problems," European Economic Review, Elsevier, vol. 46(8), pages 1369-1378, September.
    14. Uri Gneezy & John A. List & Jeffrey A. Livingston & Xiangdong Qin & Sally Sadoff & Yang Xu, 2019. "Measuring Success in Education: The Role of Effort on the Test Itself," American Economic Review: Insights, American Economic Association, vol. 1(3), pages 291-308, December.
    15. Arkady Konovalov & Ian Krajbich, 2019. "Revealed strength of preference: Inference from response times," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 14(4), pages 381-394, July.
    16. Kota Saito, 2015. "Preferences for Flexibility and Randomization under Uncertainty," American Economic Review, American Economic Association, vol. 105(3), pages 1246-1271, March.
    17. Stefano DellaVigna & John A. List & Ulrike Malmendier & Gautam Rao, 2022. "Estimating Social Preferences and Gift Exchange at Work," American Economic Review, American Economic Association, vol. 112(3), pages 1038-1074, March.
    18. Armin Falk & Anke Becker & Thomas Dohmen & Benjamin Enke & David B. Huffman & Uwe Sunde, 2017. "Global Evidence on Economic Preferences," NBER Working Papers 23943, National Bureau of Economic Research, Inc.
    19. Amador-Hidalgo, Luis & Brañas-Garza, Pablo & Espín, Antonio M. & García-Muñoz, Teresa & Hernández-Román, Ana, 2021. "Cognitive abilities and risk-taking: Errors, not preferences," European Economic Review, Elsevier, vol. 134(C).
    20. Taylor, Matthew P., 2016. "Are high-ability individuals really more tolerant of risk? A test of the relationship between risk aversion and cognitive ability," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 63(C), pages 136-147.
    21. Lau Lilleholt, 2019. "Cognitive ability and risk aversion: A systematic review and meta analysis," Judgment and Decision Making, Society for Judgment and Decision Making, vol. 14(3), pages 234-279, May.
    22. Fudenberg, Drew & Levine, David K., 2012. "Fairness, risk preferences and independence: Impossibility theorems," Journal of Economic Behavior & Organization, Elsevier, vol. 81(2), pages 606-612.
    23. Matthew Rabin & Georg Weizsacker, 2009. "Narrow Bracketing and Dominated Choices," American Economic Review, American Economic Association, vol. 99(4), pages 1508-1543, September.
    24. Adam Dominiak & Wendelin Schnedler, 2011. "Attitudes toward uncertainty and randomization: an experimental study," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 48(2), pages 289-312, October.
    25. Kota Saito, 2013. "Social Preferences under Risk: Equality of Opportunity versus Equality of Outcome," American Economic Review, American Economic Association, vol. 103(7), pages 3084-3101, December.
    26. Katherine B. Coffman & David Klinowski, 2020. "The impact of penalties for wrong answers on the gender gap in test scores," Proceedings of the National Academy of Sciences, Proceedings of the National Academy of Sciences, vol. 117(16), pages 8794-8803, April.
    27. Ola Andersson & Håkan J. Holm & Jean-Robert Tyran & Erik Wengström, 2020. "Robust inference in risk elicitation tasks," Journal of Risk and Uncertainty, Springer, vol. 61(3), pages 195-209, December.
    28. Armin Falk & Anke Becker & Thomas Dohmen & Benjamin Enke & David Huffman & Uwe Sunde, 2018. "Global Evidence on Economic Preferences," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 133(4), pages 1645-1692.
    29. Marina Agranov & Paul J Healy & Kirby Nielsen, 2023. "Stable Randomisation," The Economic Journal, Royal Economic Society, vol. 133(655), pages 2553-2579.
    30. Rustichini, Aldo & DeYoung, Colin G. & Anderson, Jon E. & Burks, Stephen V., 2016. "Toward the integration of personality theory and decision theory in explaining economic behavior: An experimental investigation," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 64(C), pages 122-137.
    31. Simone Cerreia‐Vioglio & David Dillenberger & Pietro Ortoleva, 2015. "Cautious Expected Utility and the Certainty Effect," Econometrica, Econometric Society, vol. 83, pages 693-728, March.
    32. Steven D Levitt, 2021. "Heads or Tails: The Impact of a Coin Toss on Major Life Decisions and Subsequent Happiness," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 88(1), pages 378-405.
    33. Chew, S H & Epstein, Larry G & Segal, U, 1991. "Mixture Symmetry and Quadratic Utility," Econometrica, Econometric Society, vol. 59(1), pages 139-163, January.
    34. Fabio Maccheroni & Massimo Marinacci & Aldo Rustichini, 2006. "Ambiguity Aversion, Robustness, and the Variational Representation of Preferences," Econometrica, Econometric Society, vol. 74(6), pages 1447-1498, November.
    35. Bade, Sophie, 2015. "Randomization devices and the elicitation of ambiguity-averse preferences," Journal of Economic Theory, Elsevier, vol. 159(PA), pages 221-235.
    36. Marina Agranov & Pietro Ortoleva, 2022. "Revealed Preferences for Randomization: An Overview," AEA Papers and Proceedings, American Economic Association, vol. 112, pages 426-430, May.
    37. Dwenger, Nadja & Kübler, Dorothea & Weizsäcker, Georg, 2018. "Flipping a coin: Evidence from university applications," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 167, pages 240-250.
    38. Jürgen Eichberger & Simon Grant & David Kelsey, 2016. "Randomization and dynamic consistency," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 62(3), pages 547-566, August.
    39. Shane Frederick, 2005. "Cognitive Reflection and Decision Making," Journal of Economic Perspectives, American Economic Association, vol. 19(4), pages 25-42, Fall.
    40. Iriberri, Nagore & Rey-Biel, Pedro, 2021. "Brave boys and play-it-safe girls: Gender differences in willingness to guess in a large scale natural field experiment," European Economic Review, Elsevier, vol. 131(C).
    41. Chew, S H & Epstein, Larry G, 1988. "The Law of Large Numbers and the Attractiveness of Compound Gambles," Journal of Risk and Uncertainty, Springer, vol. 1(1), pages 125-132, March.
    42. Shaowei Ke & Qi Zhang, 2020. "Randomization and Ambiguity Aversion," Econometrica, Econometric Society, vol. 88(3), pages 1159-1195, May.
    43. Zapechelnyuk, Andriy, 2015. "An axiomatization of multiple-choice test scoring," Economics Letters, Elsevier, vol. 132(C), pages 24-27.
    44. Howard Raiffa, 1961. "Risk, Ambiguity, and the Savage Axioms: Comment," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 75(4), pages 690-694.
    45. Drew Fudenberg & Ryota Iijima & Tomasz Strzalecki, 2015. "Stochastic Choice and Revealed Perturbed Utility," Econometrica, Econometric Society, vol. 83, pages 2371-2409, November.
    46. Benjamin Enke & Thomas Graeber, 2023. "Cognitive Uncertainty," The Quarterly Journal of Economics, President and Fellows of Harvard College, vol. 138(4), pages 2021-2067.
    47. Yoram Halevy, 2007. "Ellsberg Revisited: An Experimental Study," Econometrica, Econometric Society, vol. 75(2), pages 503-536, March.
    48. Diab, Dalia L. & Gillespie, Michael A. & Highhouse, Scott, 2008. "Are maximizers really unhappy? The measurement of maximizing tendency," Judgment and Decision Making, Cambridge University Press, vol. 3(5), pages 364-370, June.
    49. Daniel J. Benjamin & Sebastian A. Brown & Jesse M. Shapiro, 2013. "Who Is ‘Behavioral’? Cognitive Ability And Anomalous Preferences," Journal of the European Economic Association, European Economic Association, vol. 11(6), pages 1231-1255, December.
    50. Wakker, Peter P. & Yang, Jingni, 2019. "A powerful tool for analyzing concave/convex utility and weighting functions," Journal of Economic Theory, Elsevier, vol. 181(C), pages 143-159.
    51. Paul Feldman & John Rehbeck, 2022. "Revealing a preference for mixtures: An experimental study of risk," Quantitative Economics, Econometric Society, vol. 13(2), pages 761-786, May.
    52. Oechssler, Jörg & Rau, Hannes & Roomets, Alex, 2019. "Hedging, ambiguity, and the reversal of order axiom," Games and Economic Behavior, Elsevier, vol. 117(C), pages 380-387.
    53. Lex Borghans & Huub Meijers & Bas Ter Weel, 2008. "The Role Of Noncognitive Skills In Explaining Cognitive Test Scores," Economic Inquiry, Western Economic Association International, vol. 46(1), pages 2-12, January.
    54. Marina Agranov & Pietro Ortoleva, 2017. "Stochastic Choice and Preferences for Randomization," Journal of Political Economy, University of Chicago Press, vol. 125(1), pages 40-68.
    55. Machina, Mark J, 1985. "Stochastic Choice Functions Generated from Deterministic Preferences over Lotteries," Economic Journal, Royal Economic Society, vol. 95(379), pages 575-594, September.
    56. David Budescu & Yuanchao Bo, 2015. "Analyzing Test-Taking Behavior: Decision Theory Meets Psychometric Theory," Psychometrika, Springer;The Psychometric Society, vol. 80(4), pages 1105-1122, December.
    57. Dean, Mark & Ortoleva, Pietro, 2017. "Allais, Ellsberg, and preferences for hedging," Theoretical Economics, Econometric Society, vol. 12(1), January.
    58. Katherine Baldiga, 2014. "Gender Differences in Willingness to Guess," Management Science, INFORMS, vol. 60(2), pages 434-448, February.
    59. Cettolin, Elena & Riedl, Arno, 2019. "Revealed preferences under uncertainty: Incomplete preferences and preferences for randomization," Journal of Economic Theory, Elsevier, vol. 181(C), pages 547-585.
    60. Chew, Soo Hong & Miao, Bin & Shen, Qiang & Zhong, Songfa, 2022. "Multiple-switching behavior in choice-list elicitation of risk preference," Journal of Economic Theory, Elsevier, vol. 204(C).
    61. Jonathan de Quidt & Johannes Haushofer & Christopher Roth, 2018. "Measuring and Bounding Experimenter Demand," American Economic Review, American Economic Association, vol. 108(11), pages 3266-3302, November.
    62. Schmeidler, David, 1989. "Subjective Probability and Expected Utility without Additivity," Econometrica, Econometric Society, vol. 57(3), pages 571-587, May.
    63. Tomohito Aoyama & Nobuyuki Hanaki, 2021. "Preference for randomization and validity of random incentive system under ambiguity: An experiment," ISER Discussion Paper 1140, Institute of Social and Economic Research, The University of Osaka.
    64. Simone Cerreia-Vioglio & David Dillenberger & Pietro Ortoleva & Gil Riella, 2019. "Deliberately Stochastic," American Economic Review, American Economic Association, vol. 109(7), pages 2425-2445, July.
      • Simone Cerreia-Vioglio & David Dillenberger & Pietro Ortoleva & Gil Riella, 2012. "Deliberately Stochastic," PIER Working Paper Archive 17-013, Penn Institute for Economic Research, Department of Economics, University of Pennsylvania, revised 25 May 2017.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Chew, Soo Hong & Miao, Bin & Shen, Qiang & Zhong, Songfa, 2022. "Multiple-switching behavior in choice-list elicitation of risk preference," Journal of Economic Theory, Elsevier, vol. 204(C).
    2. Zhou, Jing, 2024. "Does correlation matter in probability matching? A laboratory investigation," Journal of Economic Behavior & Organization, Elsevier, vol. 224(C), pages 876-894.
    3. Evan M. Calford, 2026. "Randomization preference and the reversal of order axiom," Journal of Risk and Uncertainty, Springer, vol. 72(1), pages 111-135, February.
    4. Christoph Kuzmics & Brian W. Rogers & Xiannong Zhang, 2024. "Randomization advice and ambiguity aversion," Journal of Risk and Uncertainty, Springer, vol. 69(1), pages 85-104, August.
    5. Shaowei Ke & Qi Zhang, 2020. "Randomization and Ambiguity Aversion," Econometrica, Econometric Society, vol. 88(3), pages 1159-1195, May.
    6. Nicolas Eber & Patrick Roger & Tristan Roger, 2024. "Finance and intelligence: An overview of the literature," Journal of Economic Surveys, Wiley Blackwell, vol. 38(2), pages 503-554, April.
    7. Aurélien Baillon & Yoram Halevy & Chen Li, 2022. "Experimental elicitation of ambiguity attitude using the random incentive system," Experimental Economics, Springer;Economic Science Association, vol. 25(3), pages 1002-1023, June.
    8. Ghirardato, Paolo & Pennesi, Daniele, 2020. "A general theory of subjective mixtures," Journal of Economic Theory, Elsevier, vol. 188(C).
    9. König-Kersting, Christian & Kops, Christopher & Trautmann, Stefan T., 2023. "A test of (weak) certainty independence," Journal of Economic Theory, Elsevier, vol. 209(C).
    10. Yoram Halevy & David Walker-Jones & Lanny Zrill, 2023. "Difficult Decisions," Working Papers tecipa-753, University of Toronto, Department of Economics.
    11. Aurélien Baillon & Yoram Halevy & Chen Li, 2022. "Randomize at Your Own Risk: On the Observability of Ambiguity Aversion," Econometrica, Econometric Society, vol. 90(3), pages 1085-1107, May.
    12. Amador, Luis & Brañas-Garza, Pablo & Espín, Antonio M. & Garcia, Teresa & Hernández, Ana, 2019. "Consistent and inconsistent choices under uncertainty: The role of cognitive abilities," MPRA Paper 95178, University Library of Munich, Germany.
    13. Willadsen, Helene & Zaccagni, Sarah & Piovesan, Marco & Wengström, Erik, 2024. "Measures of cognitive ability and choice inconsistency," Journal of Economic Behavior & Organization, Elsevier, vol. 220(C), pages 495-506.
    14. Sean Duffy & John Smith, 2025. "An economist and a psychologist form a line: What can imperfect perception of length tell us about stochastic choice?," Theory and Decision, Springer, vol. 99(3), pages 701-734, November.
    15. Jonathan Chapman & Erik Snowberg & Stephanie Wang & Colin Camerer, 2018. "Loss Attitudes in the U.S. Population: Evidence from Dynamically Optimized Sequential Experimentation (DOSE)," CESifo Working Paper Series 7262, CESifo.
    16. Heydari, Pedram, 2024. "Regret, responsibility, and randomization: A theory of stochastic choice," Journal of Economic Theory, Elsevier, vol. 217(C).
    17. Caliari, Daniele, 2023. "Rationality is not consistency," Discussion Papers, Research Unit: Economics of Change SP II 2023-304, WZB Berlin Social Science Center.
    18. Campara, Jessica & Da Costa, Newton & Matsushita, Raul & Da Silva, Sergio, 2021. "Two selves and two minds in a longitudinal survey of risk attitudes," Journal of Behavioral and Experimental Finance, Elsevier, vol. 29(C).
    19. Benjamin Monet & Vassili Vergopoulos, 2024. "Ambiguity, randomization and the timing of resolution of uncertainty," Economic Theory, Springer;Society for the Advancement of Economic Theory (SAET), vol. 78(4), pages 1021-1045, December.
    20. Sara Arts & Qiyan Ong & Jianying Qiu, 2024. "Measuring decision confidence," Experimental Economics, Springer;Economic Science Association, vol. 27(3), pages 582-603, July.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    JEL classification:

    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • D81 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Criteria for Decision-Making under Risk and Uncertainty

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:jeborg:v:237:y:2025:i:c:s0167268125003038. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/jebo .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.