IDEAS home Printed from https://ideas.repec.org/p/rco/dpaper/562.html

A Horserace of Methods for Eliciting Induced Beliefs Online

Author

Listed:
  • Daniel Banko-Ferran

    (University of Pittsburgh)

  • Valeria Burdea

    (LMU Munich)

  • Jonathan Woon

    (University of Pittsburgh)

Abstract

This study evaluates the effectiveness of three widely used belief elicitation methods in an online setting: the binarized scoring rule (BSR), the stochastic Becker-DeGroot-Marschak mechanism (BDM), and unincentivized introspection. Despite the theoretical advantages of incentive-compatible methods (BSR and BDM), we find that they impose significantly higher cognitive costs on participants, requiring more time and effort to implement, without delivering clear improvements in belief accuracy. In fact, BSR systematically leads to greater errors in reported beliefs compared to introspection, while BDM also reduces accuracy, though to a lesser extent. Surprisingly, individual differences in probabilistic reasoning skills do not mitigate these errors for BSR but do help improve accuracy under BDM. Our findings suggest that simpler, unincentivized approaches may offer comparable or even superior accuracy at a lower cognitive cost. These results have broad implications for the design of experiments and the interpretation of belief data in behavioral and experimental economics.

Suggested Citation

  • Daniel Banko-Ferran & Valeria Burdea & Jonathan Woon, 2026. "A Horserace of Methods for Eliciting Induced Beliefs Online," Rationality and Competition Discussion Paper Series 562, CRC TRR 190 Rationality and Competition.
    • Handle: RePEc:rco:dpaper:562
    as

    Download full text from publisher

    File URL: https://rationality-and-competition.de/wp-content/uploads/discussion_paper/562.pdf
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Smith, Vernon L & Walker, James M, 1993. "Monetary Rewards and Decision Cost in Experimental Economics," Economic Inquiry, Western Economic Association International, vol. 31(2), pages 245-261, April.
    2. Markus M. Möbius & Muriel Niederle & Paul Niehaus & Tanya S. Rosenblat, 2022. "Managing Self-Confidence: Theory and Experimental Evidence," Management Science, INFORMS, vol. 68(11), pages 7793-7817, November.
    3. Holt, Charles A. & Smith, Angela M., 2009. "An update on Bayesian updating," Journal of Economic Behavior & Organization, Elsevier, vol. 69(2), pages 125-134, February.
    4. Harrison, Glenn W. & Martínez-Correa, Jimmy & Swarthout, J. Todd, 2014. "Eliciting subjective probabilities with binary lotteries," Journal of Economic Behavior & Organization, Elsevier, vol. 101(C), pages 128-140.
    5. Andrew Schotter & Isabel Trevino, 2014. "Belief Elicitation in the Laboratory," Annual Review of Economics, Annual Reviews, vol. 6(1), pages 103-128, August.
    6. Edi Karni, 2009. "A Mechanism for Eliciting Probabilities," Econometrica, Econometric Society, vol. 77(2), pages 603-606, March.
    7. Tanjim Hossain & Ryo Okui, 2013. "The Binarized Scoring Rule," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 80(3), pages 984-1001.
    8. Charles F. Manski, 2004. "Measuring Expectations," Econometrica, Econometric Society, vol. 72(5), pages 1329-1376, September.
    9. James Banovetz & Ryan Oprea, 2023. "Complexity and Procedural Choice," American Economic Journal: Microeconomics, American Economic Association, vol. 15(2), pages 384-413, May.
    10. Guillaume Hollard & Sébastien Massoni & Jean-Christophe Vergnaud, 2016. "In search of good probability assessors: an experimental comparison of elicitation rules for confidence judgments," Theory and Decision, Springer, vol. 80(3), pages 363-387, March.
    11. Charness, Gary & Gneezy, Uri & Rasocha, Vlastimil, 2021. "Experimental methods: Eliciting beliefs," Journal of Economic Behavior & Organization, Elsevier, vol. 189(C), pages 234-256.
    12. Guillaume Hollard & Sébastien Massoni & Jean-Christophe Vergnaud, 2016. "In search of good probability assessors: an experimental comparison of elicitation rules for confidence judgments," Université Paris1 Panthéon-Sorbonne (Post-Print and Working Papers) hal-01306258, HAL.
    13. Franklin Allen, 1987. "Notes--Discovering Personal Probabilities When Utility Functions are Unknown," Management Science, INFORMS, vol. 33(4), pages 542-544, April.
    14. Ingrid Burfurd & Tom Wilkening, 2018. "Experimental guidance for eliciting beliefs with the Stochastic Becker–DeGroot–Marschak mechanism," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 4(1), pages 15-28, July.
    15. Wang, Stephanie W., 2011. "Incentive effects: The case of belief elicitation from individuals in groups," Economics Letters, Elsevier, vol. 111(1), pages 30-33, April.
    16. Aurélien Baillon & Han Bleichrodt, 2015. "Testing Ambiguity Models through the Measurement of Probabilities for Gains and Losses," American Economic Journal: Microeconomics, American Economic Association, vol. 7(2), pages 77-100, May.
    17. Grether, David M., 1992. "Testing bayes rule and the representativeness heuristic: Some experimental evidence," Journal of Economic Behavior & Organization, Elsevier, vol. 17(1), pages 31-57, January.
    18. Plott, Charles R, 1986. "Rational Choice in Experimental Markets," The Journal of Business, University of Chicago Press, vol. 59(4), pages 301-327, October.
    19. Jindrich Matousek & Tomas Havranek & Zuzana Irsova, 2022. "Individual discount rates: a meta-analysis of experimental evidence," Experimental Economics, Springer;Economic Science Association, vol. 25(1), pages 318-358, February.
    20. Li Hao & Daniel Houser, 2012. "Belief elicitation in the presence of naïve respondents: An experimental study," Journal of Risk and Uncertainty, Springer, vol. 44(2), pages 161-180, April.
    21. David Danz & Lise Vesterlund & Alistair J. Wilson, 2022. "Belief Elicitation and Behavioral Incentive Compatibility," American Economic Review, American Economic Association, vol. 112(9), pages 2851-2883, September.
    22. Karl Schlag & James Tremewan, 2021. "Simple belief elicitation: An experimental evaluation," Journal of Risk and Uncertainty, Springer, vol. 62(2), pages 137-155, April.
    23. Erkal, Nisvan & Gangadharan, Lata & Koh, Boon Han, 2020. "Replication: Belief elicitation with quadratic and binarized scoring rules," Journal of Economic Psychology, Elsevier, vol. 81(C).
    24. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    25. Emanuel Vespa & Alistair J. Wilson, 2016. "Communication with multiple senders: An experiment," Quantitative Economics, Econometric Society, vol. 7(1), pages 1-36, March.
    26. Charles A. Holt & Angela M. Smith, 2016. "Belief Elicitation with a Synchronized Lottery Choice Menu That Is Invariant to Risk Attitudes," American Economic Journal: Microeconomics, American Economic Association, vol. 8(1), pages 110-139, February.
    27. Burdea, Valeria & Woon, Jonathan, 2022. "Online belief elicitation methods," Journal of Economic Psychology, Elsevier, vol. 90(C).
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Charness, Gary & Gneezy, Uri & Rasocha, Vlastimil, 2021. "Experimental methods: Eliciting beliefs," Journal of Economic Behavior & Organization, Elsevier, vol. 189(C), pages 234-256.
    2. Burdea, Valeria & Woon, Jonathan, 2022. "Online belief elicitation methods," Journal of Economic Psychology, Elsevier, vol. 90(C).
    3. Ingrid Burfurd & Tom Wilkening, 2022. "Cognitive heterogeneity and complex belief elicitation," Experimental Economics, Springer;Economic Science Association, vol. 25(2), pages 557-592, April.
    4. Gangadharan, Lata & Grossman, Philip J. & Xue, Nina, 2024. "Belief elicitation under competing motivations: Does it matter how you ask?," European Economic Review, Elsevier, vol. 169(C).
    5. Stephen L. Cheung & Lachlan Johnstone, 2025. "True overconfidence, revealed through actions: An experiment," Journal of Risk and Uncertainty, Springer, vol. 70(2), pages 171-199, April.
    6. Dominik Bauer & Irenaeus Wolff, 2018. "Biases in Beliefs: Experimental Evidence," TWI Research Paper Series 109, Thurgauer Wirtschaftsinstitut, Universität Konstanz.
    7. Guillaume Hollard & Sébastien Massoni & Jean-Christophe Vergnaud, 2016. "In search of good probability assessors: an experimental comparison of elicitation rules for confidence judgments," Theory and Decision, Springer, vol. 80(3), pages 363-387, March.
    8. Folli, Dominik & Wolff, Irenaeus, 2022. "Biases in belief reports," Journal of Economic Psychology, Elsevier, vol. 88(C).
    9. Bauer, Dominik & Wolff, Irenaeus, 2019. "Biases in Beliefs," VfS Annual Conference 2019 (Leipzig): 30 Years after the Fall of the Berlin Wall - Democracy and Market Economy 203601, Verein für Socialpolitik / German Economic Association.
    10. Ingrid Burfurd & Tom Wilkening, 2018. "Experimental guidance for eliciting beliefs with the Stochastic Becker–DeGroot–Marschak mechanism," Journal of the Economic Science Association, Springer;Economic Science Association, vol. 4(1), pages 15-28, July.
    11. Markus M. Möbius & Muriel Niederle & Paul Niehaus & Tanya S. Rosenblat, 2022. "Managing Self-Confidence: Theory and Experimental Evidence," Management Science, INFORMS, vol. 68(11), pages 7793-7817, November.
    12. Lata Gangadharan & Philip J. Grossman & Nina Xue, 2022. "Stepping Stone: Identifying self-image concerns from motivated beliefs: Does it matter how and whom you ask?," Monash Economics Working Papers 2022-05, Monash University, Department of Economics.
    13. Ersoy, Fulya Y., 2025. "Do incentives matter in elicitation of beliefs?," Journal of Behavioral and Experimental Economics (formerly The Journal of Socio-Economics), Elsevier, vol. 119(C).
    14. Juan Dubra & Jean-Pierre Beno t & Giorgia Romagnoli, 2019. "Belief elicitation when more than money matters," Documentos de Trabajo/Working Papers 1901, Facultad de Ciencias Empresariales y Economia. Universidad de Montevideo..
    15. Lata Gangadharan & Philip J. Grossman & Nina Xue, 2021. "Identifying self-image concerns from motivated beliefs: Does it matter how and whom you ask?," Monash Economics Working Papers 2021-17, Monash University, Department of Economics.
    16. Li Hao & Daniel Houser, 2012. "Belief elicitation in the presence of naïve respondents: An experimental study," Journal of Risk and Uncertainty, Springer, vol. 44(2), pages 161-180, April.
    17. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    18. Eyting, Markus & Schmidt, Patrick, 2021. "Belief elicitation with multiple point predictions," European Economic Review, Elsevier, vol. 135(C).
    19. Crosetto, Paolo & Filippin, Antonio & Katuščák, Peter & Smith, John, 2020. "Central tendency bias in belief elicitation," Journal of Economic Psychology, Elsevier, vol. 78(C).
    20. Jean-Pierre Benoît & Juan Dubra & Giorgia Romagnoli, 2022. "Belief Elicitation When More than Money Matters: Controlling for "Control"," American Economic Journal: Microeconomics, American Economic Association, vol. 14(3), pages 837-888, August.

    More about this item

    Keywords

    ;
    ;
    ;
    ;

    JEL classification:

    • C81 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Methodology for Collecting, Estimating, and Organizing Microeconomic Data; Data Access
    • C89 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Other
    • D83 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Search; Learning; Information and Knowledge; Communication; Belief; Unawareness
    • D91 - Microeconomics - - Micro-Based Behavioral Economics - - - Role and Effects of Psychological, Emotional, Social, and Cognitive Factors on Decision Making

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:rco:dpaper:562. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Viviana Lalli (email available below). General contact details of provider: https://rationality-and-competition.de .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.