IDEAS home Printed from https://ideas.repec.org/p/vie/viennp/vie1401.html
   My bibliography  Save this paper

A Penny for your Thoughts: A Survey of Methods of Eliciting Beliefs

Author

Abstract

Incentivized methods for eliciting subjective probabilities in economic experiments present the subject with risky choices or bets that encourage truthful reporting. We discuss the most prominent elicitation methods and their underlying assumptions, provide theoretical comparisons, and propose some extensions to the standard framework. In addition, we survey the empirical literature on the performance of these elicitation methods in actual experiments, considering also practical issues of implementation such as order effects, hedging, and different ways of presenting probabilities and payment schemes to experimental subjects. We end with some thoughts on the merits of using incentives for belief elicitation and some guidelines for implementation.

Suggested Citation

  • Karl Schlag & James Tremewan & Joel von der Weele, 2014. "A Penny for your Thoughts: A Survey of Methods of Eliciting Beliefs," Vienna Economics Papers vie1401, University of Vienna, Department of Economics.
    • Handle: RePEc:vie:viennp:vie1401
    as

    Download full text from publisher

    File URL: https://papersecon.univie.ac.at/RePEc/vie/viennp/vie1401.pdf
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. Manski, Charles F., 2006. "Interpreting the predictions of prediction markets," Economics Letters, Elsevier, vol. 91(3), pages 425-429, June.
    2. Karl Schlag & James Tremevan, 2012. "Simple Belief Elicitation," Vienna Economics Papers 1204, University of Vienna, Department of Economics.
    3. Armantier, Olivier & Treich, Nicolas, 2013. "Eliciting beliefs: Proper scoring rules, incentives, stakes and hedging," European Economic Review, Elsevier, vol. 62(C), pages 17-40.
    4. Terrance Hurley & Jason Shogren, 2005. "An Experimental Comparison of Induced and Elicited Beliefs," Journal of Risk and Uncertainty, Springer, vol. 30(2), pages 169-188, January.
    5. Rutström, E. Elisabet & Wilcox, Nathaniel T., 2009. "Stated beliefs versus inferred beliefs: A methodological inquiry and experimental test," Games and Economic Behavior, Elsevier, vol. 67(2), pages 616-632, November.
    6. Guillaume Hollard & Sébastien Massoni & Jean-Christophe Vergnaud, 2010. "Subjective beliefs formation and elicitation rules : experimental evidence," Post-Print halshs-00543828, HAL.
    7. Steffen Andersen & John Fountain & Glenn Harrison & E. Rutström, 2014. "Estimating subjective probabilities," Journal of Risk and Uncertainty, Springer, vol. 48(3), pages 207-229, June.
    8. Mariana Blanco & Dirk Engelmann & Alexander Koch & Hans-Theo Normann, 2010. "Belief elicitation in experiments: is there a hedging problem?," Experimental Economics, Springer;Economic Science Association, vol. 13(4), pages 412-438, December.
    9. Reinhard Selten & Abdolkarim Sadrieh & Klaus Abbink, 1999. "Money Does Not Induce Risk Neutral Behavior, but Binary Lotteries Do even Worse," Theory and Decision, Springer, vol. 46(3), pages 213-252, June.
    10. Stefan T. Trautmann & Gijs Kuilen, 2015. "Belief Elicitation: A Horse Race among Truth Serums," Economic Journal, Royal Economic Society, vol. 125(589), pages 2116-2135, December.
    11. James E. Matheson & Robert L. Winkler, 1976. "Scoring Rules for Continuous Probability Distributions," Management Science, INFORMS, vol. 22(10), pages 1087-1096, June.
    12. Daniel Friedman, 1983. "Effective Scoring Rules for Probabilistic Forecasts," Management Science, INFORMS, vol. 29(4), pages 447-454, April.
    13. Camerer, Colin F & Hogarth, Robin M, 1999. "The Effects of Financial Incentives in Experiments: A Review and Capital-Labor-Production Framework," Journal of Risk and Uncertainty, Springer, vol. 19(1-3), pages 7-42, December.
    14. Frank Heinemann & Rosemarie Nagel & Peter Ockenfels, 2009. "Measuring Strategic Uncertainty in Coordination Games," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 76(1), pages 181-221.
    15. Harrison, Glenn W. & Martínez-Correa, Jimmy & Swarthout, J. Todd, 2014. "Eliciting subjective probabilities with binary lotteries," Journal of Economic Behavior & Organization, Elsevier, vol. 101(C), pages 128-140.
    16. Andrew Schotter & Isabel Trevino, 2014. "Belief Elicitation in the Laboratory," Annual Review of Economics, Annual Reviews, vol. 6(1), pages 103-128, August.
    17. Nolan Miller & Paul Resnick & Richard Zeckhauser, 2005. "Eliciting Informative Feedback: The Peer-Prediction Method," Management Science, INFORMS, vol. 51(9), pages 1359-1373, September.
    18. Edi Karni, 2009. "A Mechanism for Eliciting Probabilities," Econometrica, Econometric Society, vol. 77(2), pages 603-606, March.
    19. Tanjim Hossain & Ryo Okui, 2013. "The Binarized Scoring Rule," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 80(3), pages 984-1001.
    20. Simon Gächter & Elke Renner, 2010. "The effects of (incentivized) belief elicitation in public goods experiments," Experimental Economics, Springer;Economic Science Association, vol. 13(3), pages 364-377, September.
    21. Koessler, Frédéric & Noussair, Charles & Ziegelmeyer, Anthony, 2012. "Information aggregation and belief elicitation in experimental parimutuel betting markets," Journal of Economic Behavior & Organization, Elsevier, vol. 83(2), pages 195-208.
    22. Charles F. Manski, 2004. "Measuring Expectations," Econometrica, Econometric Society, vol. 72(5), pages 1329-1376, September.
    23. John Fountain & Glenn Harrison, 2011. "What do prediction markets predict?," Applied Economics Letters, Taylor & Francis Journals, vol. 18(3), pages 267-272.
    24. Schmalensee, Richard, 1976. "An Experimental Study of Expectation Formation," Econometrica, Econometric Society, vol. 44(1), pages 17-41, January.
    25. Harrison, Glenn W. & Martínez-Correa, Jimmy & Swarthout, J. Todd, 2013. "Inducing risk neutral preferences with binary lotteries: A reconsideration," Journal of Economic Behavior & Organization, Elsevier, vol. 94(C), pages 145-159.
    26. Franklin Allen, 1987. "Notes--Discovering Personal Probabilities When Utility Functions are Unknown," Management Science, INFORMS, vol. 33(4), pages 542-544, April.
    27. Price, Paul C., 1998. "Effects of a Relative-Frequency Elicitation Question on Likelihood Judgment Accuracy: The Case of External Correspondence, , , , ," Organizational Behavior and Human Decision Processes, Elsevier, vol. 76(3), pages 277-297, December.
    28. Palfrey, Thomas R. & Wang, Stephanie W., 2009. "On eliciting beliefs in strategic games," Journal of Economic Behavior & Organization, Elsevier, vol. 71(2), pages 98-109, August.
    29. Charles A. Holt & Susan K. Laury, 2002. "Risk Aversion and Incentive Effects," American Economic Review, American Economic Association, vol. 92(5), pages 1644-1655, December.
    30. Emir Shuford & Arthur Albert & H. Edward Massengill, 1966. "Admissible probability measurement procedures," Psychometrika, Springer;The Psychometric Society, vol. 31(2), pages 125-145, June.
    31. Wang, Stephanie W., 2011. "Incentive effects: The case of belief elicitation from individuals in groups," Economics Letters, Elsevier, vol. 111(1), pages 30-33, April.
    32. Manski, Charles F., 2002. "Identification of decision rules in experiments on simple games of proposal and response," European Economic Review, Elsevier, vol. 46(4-5), pages 880-891, May.
    33. Berg, Joyce E. & Rietz, Thomas A. & Dickhaut, John W., 2008. "On the Performance of the Lottery Procedure for Controlling Risk Preferences," Handbook of Experimental Economics Results, in: Charles R. Plott & Vernon L. Smith (ed.), Handbook of Experimental Economics Results, edition 1, volume 1, chapter 115, pages 1087-1097, Elsevier.
    34. Michał Krawczyk, 2011. "Overconfident for real? Proper scoring for confidence intervals," Working Papers 2011-15, Faculty of Economic Sciences, University of Warsaw.
    35. Costa-Gomes, Miguel A. & Huck, Steffen & Weizsäcker, Georg, 2014. "Beliefs and actions in the trust game: Creating instrumental variables to estimate the causal effect," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 88, pages 298-309.
    36. Gilboa, Itzhak, 1987. "Expected utility with purely subjective non-additive probabilities," Journal of Mathematical Economics, Elsevier, vol. 16(1), pages 65-88, February.
    37. Dirk Engelmann & Martin Strobel, 2000. "The False Consensus Effect Disappears if Representative Information and Monetary Incentives Are Given," Experimental Economics, Springer;Economic Science Association, vol. 3(3), pages 241-260, December.
    38. Wolfers, Justin & Zitzewitz, Eric, 2006. "Interpreting Prediction Market Prices as Probabilities," IZA Discussion Papers 2092, Institute of Labor Economics (IZA).
    39. Alexander Smith, 2013. "Estimating the causal effect of beliefs on contributions in repeated public good games," Experimental Economics, Springer;Economic Science Association, vol. 16(3), pages 414-425, September.
    40. Erev, Ido & Bornstein, Gary & Wallsten, Thomas S., 1993. "The Negative Effect of Probability Assessments on Decision Quality," Organizational Behavior and Human Decision Processes, Elsevier, vol. 55(1), pages 78-94, June.
    41. repec:exc:wpaper:2013-05 is not listed on IDEAS
    42. Isaac M. Lipkus & Greg Samsa & Barbara K. Rimer, 2001. "General Performance on a Numeracy Scale among Highly Educated Samples," Medical Decision Making, , vol. 21(1), pages 37-44, February.
    43. Jaffray, Jean-Yves & Karni, Edi, 1999. "Elicitation of Subjective Probabilities When the Initial Endowment is Unobservable," Journal of Risk and Uncertainty, Springer, vol. 18(1), pages 5-20, April.
    44. Huck, Steffen & Weizsacker, Georg, 2002. "Do players correctly estimate what others do? : Evidence of conservatism in beliefs," Journal of Economic Behavior & Organization, Elsevier, vol. 47(1), pages 71-85, January.
    45. Glenn Harrison & E. Rutström, 2009. "Expected utility theory and prospect theory: one wedding and a decent funeral," Experimental Economics, Springer;Economic Science Association, vol. 12(2), pages 133-158, June.
    46. Nathaniel T. Wilcox & Nick Feltovich, 2000. "Thinking Like a Game Theorist: Comment," Monash Economics Working Papers archive-30, Monash University, Department of Economics.
    47. Yaw Nyarko & Andrew Schotter, 2002. "An Experimental Study of Belief Learning Using Elicited Beliefs," Econometrica, Econometric Society, vol. 70(3), pages 971-1005, May.
    48. Croson, Rachel T. A., 2000. "Thinking like a game theorist: factors affecting the frequency of equilibrium play," Journal of Economic Behavior & Organization, Elsevier, vol. 41(3), pages 299-314, March.
    49. R. Winkler & Javier Muñoz & José Cervera & José Bernardo & Gail Blattenberger & Joseph Kadane & Dennis Lindley & Allan Murphy & Robert Oliver & David Ríos-Insua, 1996. "Scoring rules and the evaluation of probabilities," TEST: An Official Journal of the Spanish Society of Statistics and Operations Research, Springer;Sociedad de Estadística e Investigación Operativa, vol. 5(1), pages 1-60, June.
    50. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    51. McKelvey, Richard D & Page, Talbot, 1990. "Public and Private Information: An Experimental Study of Information Pooling," Econometrica, Econometric Society, vol. 58(6), pages 1321-1339, November.
    52. Karni, Edi, 1999. "Elicitation of Subjective Probabilities When Preferences Are State-Dependent," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 40(2), pages 479-486, May.
    53. Hoffmann, Timo, 2014. "The Effect of Belief Elicitation Game Play," VfS Annual Conference 2014 (Hamburg): Evidence-based Economic Policy 100483, Verein für Socialpolitik / German Economic Association.
    54. Delavande, Adeline & Giné, Xavier & McKenzie, David, 2011. "Measuring subjective expectations in developing countries: A critical review and new evidence," Journal of Development Economics, Elsevier, vol. 94(2), pages 151-163, March.
    55. Victor Richmond R. Jose & Robert L. Winkler, 2009. "Evaluating Quantile Assessments," Operations Research, INFORMS, vol. 57(5), pages 1287-1297, October.
    56. Garthwaite, Paul H. & Kadane, Joseph B. & O'Hagan, Anthony, 2005. "Statistical Methods for Eliciting Probability Distributions," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 680-701, June.
    57. Offerman, Theo & Sonnemans, Joep & Schram, Arthur, 1996. "Value Orientations, Expectations and Voluntary Contributions in Public Goods," Economic Journal, Royal Economic Society, vol. 106(437), pages 817-845, July.
    58. Harrison, Glenn W. & Martínez-Correa, Jimmy & Swarthout, J. Todd & Ulm, Eric R., 2017. "Scoring rules for subjective probability distributions," Journal of Economic Behavior & Organization, Elsevier, vol. 134(C), pages 430-448.
    59. Reinhard Selten, 1998. "Axiomatic Characterization of the Quadratic Scoring Rule," Experimental Economics, Springer;Economic Science Association, vol. 1(1), pages 43-61, June.
    60. Guerra, Gerardo & John Zizzo, Daniel, 2004. "Trust responsiveness and beliefs," Journal of Economic Behavior & Organization, Elsevier, vol. 55(1), pages 25-30, September.
    61. Florian Artinger & Filippos Exadaktylos & Hannes Koppel & Lauri Sääksvuori, 2010. "Applying Quadratic Scoring Rule transparently in multiple choice settings: A note," ThE Papers 10/01, Department of Economic Theory and Economic History of the University of Granada..
    62. Machina, Mark J & Schmeidler, David, 1992. "A More Robust Definition of Subjective Probability," Econometrica, Econometric Society, vol. 60(4), pages 745-780, July.
    63. Ramsey, Frank P., 1926. "Truth and Probability," Histoy of Economic Thought Chapters, in: Braithwaite, R. B. (ed.),The Foundations of Mathematics and other Logical Essays, chapter 7, pages 156-198, McMaster University Archive for the History of Economic Thought.
    64. Olivier Armantier & Nicolas Treich, 2009. "Subjective Probabilities In Games: An Application To The Overbidding Puzzle," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 50(4), pages 1079-1102, November.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    2. Eyting, Markus & Schmidt, Patrick, 2021. "Belief elicitation with multiple point predictions," European Economic Review, Elsevier, vol. 135(C).
    3. Charness, Gary & Gneezy, Uri & Rasocha, Vlastimil, 2021. "Experimental methods: Eliciting beliefs," Journal of Economic Behavior & Organization, Elsevier, vol. 189(C), pages 234-256.
    4. Armantier, Olivier & Treich, Nicolas, 2013. "Eliciting beliefs: Proper scoring rules, incentives, stakes and hedging," European Economic Review, Elsevier, vol. 62(C), pages 17-40.
    5. Mariana Blanco & Dirk Engelmann & Alexander Koch & Hans-Theo Normann, 2010. "Belief elicitation in experiments: is there a hedging problem?," Experimental Economics, Springer;Economic Science Association, vol. 13(4), pages 412-438, December.
    6. Steffen Andersen & John Fountain & Glenn Harrison & E. Rutström, 2014. "Estimating subjective probabilities," Journal of Risk and Uncertainty, Springer, vol. 48(3), pages 207-229, June.
    7. Folli, Dominik & Wolff, Irenaeus, 2022. "Biases in belief reports," Journal of Economic Psychology, Elsevier, vol. 88(C).
    8. Tsakas, Elias, 2018. "Robust scoring rules," Research Memorandum 023, Maastricht University, Graduate School of Business and Economics (GSBE).
    9. Markus Eyting & Patrick Schmidt, 2019. "Belief Elicitation with Multiple Point Predictions," Working Papers 1818, Gutenberg School of Management and Economics, Johannes Gutenberg-Universität Mainz, revised 16 Nov 2020.
    10. Costa-Gomes, Miguel A. & Huck, Steffen & Weizsäcker, Georg, 2014. "Beliefs and actions in the trust game: Creating instrumental variables to estimate the causal effect," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 88, pages 298-309.
    11. Stefan T. Trautmann & Gijs Kuilen, 2015. "Belief Elicitation: A Horse Race among Truth Serums," Economic Journal, Royal Economic Society, vol. 125(589), pages 2116-2135, December.
    12. Li Hao & Daniel Houser, 2012. "Belief elicitation in the presence of naïve respondents: An experimental study," Journal of Risk and Uncertainty, Springer, vol. 44(2), pages 161-180, April.
    13. Harrison, Glenn W. & Martínez-Correa, Jimmy & Swarthout, J. Todd, 2014. "Eliciting subjective probabilities with binary lotteries," Journal of Economic Behavior & Organization, Elsevier, vol. 101(C), pages 128-140.
    14. Sandra Ludwig & Julia Nafziger, 2011. "Beliefs about overconfidence," Theory and Decision, Springer, vol. 70(4), pages 475-500, April.
    15. Dominik Bauer & Irenaeus Wolff, 2018. "Biases in Beliefs: Experimental Evidence," TWI Research Paper Series 109, Thurgauer Wirtschaftsinstitut, Universität Konstanz.
    16. Bauer, Dominik & Wolff, Irenaeus, 2019. "Biases in Beliefs," VfS Annual Conference 2019 (Leipzig): 30 Years after the Fall of the Berlin Wall - Democracy and Market Economy 203601, Verein für Socialpolitik / German Economic Association.
    17. Blanco, Mariana & Engelmann, Dirk & Koch, Alexander K. & Normann, Hans-Theo, 2014. "Preferences and beliefs in a sequential social dilemma: a within-subjects analysis," Games and Economic Behavior, Elsevier, vol. 87(C), pages 122-135.
    18. Theo Offerman & Asa B. Palley, 2016. "Lossed in translation: an off-the-shelf method to recover probabilistic beliefs from loss-averse agents," Experimental Economics, Springer;Economic Science Association, vol. 19(1), pages 1-30, March.
    19. Burdea, Valeria & Woon, Jonathan, 2022. "Online belief elicitation methods," Journal of Economic Psychology, Elsevier, vol. 90(C).
    20. Crosetto, Paolo & Filippin, Antonio & Katuščák, Peter & Smith, John, 2020. "Central tendency bias in belief elicitation," Journal of Economic Psychology, Elsevier, vol. 78(C).

    More about this item

    JEL classification:

    • C83 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Survey Methods; Sampling Methods
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior
    • D83 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Search; Learning; Information and Knowledge; Communication; Belief; Unawareness

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:vie:viennp:vie1401. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Paper Administrator (email available below). General contact details of provider: https://econ.univie.ac.at/ .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.