IDEAS home Printed from https://ideas.repec.org/a/eee/eecrev/v62y2013icp17-40.html
   My bibliography  Save this article

Eliciting beliefs: Proper scoring rules, incentives, stakes and hedging

Author

Listed:
  • Armantier, Olivier
  • Treich, Nicolas

Abstract

Proper Scoring Rules (PSRs) are popular incentivized mechanisms to elicit an agent's beliefs. This paper combines theory and experiment to characterize how PSRs bias reported beliefs when (i) the PSR payments are increased, (ii) the agent has a financial stake in the event she is predicting, and (iii) the agent can hedge her prediction by taking an additional action. In contrast with previous literature, the PSR biases are characterized for all PSRs and all risk averse agents. Our results reveal complex distortions of reported beliefs, thereby raising concerns about the ability of PSRs to recover truthful beliefs in general decision-making environments.

Suggested Citation

  • Armantier, Olivier & Treich, Nicolas, 2013. "Eliciting beliefs: Proper scoring rules, incentives, stakes and hedging," European Economic Review, Elsevier, vol. 62(C), pages 17-40.
    • Handle: RePEc:eee:eecrev:v:62:y:2013:i:c:p:17-40
    DOI: 10.1016/j.euroecorev.2013.03.008
    as

    Download full text from publisher

    File URL: http://www.sciencedirect.com/science/article/pii/S001429211300041X
    Download Restriction: Full text for ScienceDirect subscribers only
    File URL: https://libkey.io/10.1016/j.euroecorev.2013.03.008?utm_source=ideas
    LibKey link: if access is restricted and if your library uses this service, LibKey will redirect you to where you can use your library subscription to access this item
    ---><---

    As the access to this document is restricted, you may want to look for a different version below or search for a different version of it.

    Other versions of this item:

    References listed on IDEAS

    as
    1. Terrance Hurley & Jason Shogren, 2005. "An Experimental Comparison of Induced and Elicited Beliefs," Journal of Risk and Uncertainty, Springer, vol. 30(2), pages 169-188, January.
    2. Rutström, E. Elisabet & Wilcox, Nathaniel T., 2009. "Stated beliefs versus inferred beliefs: A methodological inquiry and experimental test," Games and Economic Behavior, Elsevier, vol. 67(2), pages 616-632, November.
    3. Steffen Andersen & John Fountain & Glenn Harrison & E. Rutström, 2014. "Estimating subjective probabilities," Journal of Risk and Uncertainty, Springer, vol. 48(3), pages 207-229, June.
    4. Reinhard Selten & Abdolkarim Sadrieh & Klaus Abbink, 1999. "Money Does Not Induce Risk Neutral Behavior, but Binary Lotteries Do even Worse," Theory and Decision, Springer, vol. 46(3), pages 213-252, June.
    5. Davis, Douglas D. & Holt, Charles a., 1993. "Experimental economics: Methods, problems and promise," Estudios Económicos, El Colegio de México, Centro de Estudios Económicos, vol. 8(2), pages 179-212.
    6. Miguel A. Costa-Gomes & Georg Weizsäcker, 2008. "Stated Beliefs and Play in Normal-Form Games," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 75(3), pages 729-762.
    7. Alan S. Blinder & Michael Ehrmann & Marcel Fratzscher & Jakob De Haan & David-Jan Jansen, 2008. "Central Bank Communication and Monetary Policy: A Survey of Theory and Evidence," Journal of Economic Literature, American Economic Association, vol. 46(4), pages 910-945, December.
    8. Edi Karni, 2009. "A Mechanism for Eliciting Probabilities," Econometrica, Econometric Society, vol. 77(2), pages 603-606, March.
    9. Simon Gächter & Elke Renner, 2010. "The effects of (incentivized) belief elicitation in public goods experiments," Experimental Economics, Springer;Economic Science Association, vol. 13(3), pages 364-377, September.
    10. Osband, Kent, 1989. "Optimal Forecasting Incentives," Journal of Political Economy, University of Chicago Press, vol. 97(5), pages 1091-1112, October.
    11. Charles N. Noussair & Stefan T. Trautmann & Gijs van de Kuilen, 2014. "Higher Order Risk Attitudes, Demographics, and Financial Decisions," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 81(1), pages 325-355.
    12. Palfrey, Thomas R. & Wang, Stephanie W., 2009. "On eliciting beliefs in strategic games," Journal of Economic Behavior & Organization, Elsevier, vol. 71(2), pages 98-109, August.
    13. Charles A. Holt & Susan K. Laury, 2002. "Risk Aversion and Incentive Effects," American Economic Review, American Economic Association, vol. 92(5), pages 1644-1655, December.
    14. Hans-Martin von Gaudecker & Arthur van Soest & Erik Wengstrom, 2011. "Heterogeneity in Risky Choice Behavior in a Broad Population," American Economic Review, American Economic Association, vol. 101(2), pages 664-694, April.
    15. Marco Ottaviani & Peter Norman Sørensen, 2007. "Outcome Manipulation in Corporate Prediction Markets," Journal of the European Economic Association, MIT Press, vol. 5(2-3), pages 554-563, 04-05.
    16. David Danz & Dietmar Fehr & Dorothea Kübler, 2012. "Information and beliefs in a repeated normal-form game," Experimental Economics, Springer;Economic Science Association, vol. 15(4), pages 622-640, December.
    17. Urs Fischbacher & Simon Gachter, 2010. "Social Preferences, Beliefs, and the Dynamics of Free Riding in Public Goods Experiments," American Economic Review, American Economic Association, vol. 100(1), pages 541-556, March.
    18. Jaffray, Jean-Yves & Karni, Edi, 1999. "Elicitation of Subjective Probabilities When the Initial Endowment is Unobservable," Journal of Risk and Uncertainty, Springer, vol. 18(1), pages 5-20, April.
    19. Huck, Steffen & Weizsacker, Georg, 2002. "Do players correctly estimate what others do? : Evidence of conservatism in beliefs," Journal of Economic Behavior & Organization, Elsevier, vol. 47(1), pages 71-85, January.
    20. Alan S. Blinder & Michael Ehrmann & Marcel Fratzscher & Jakob De Haan & David-Jan Jansen, 2008. "Central Bank Communication and Monetary Policy: A Survey of Theory and Evidence," Journal of Economic Literature, American Economic Association, vol. 46(4), pages 910-945, December.
    21. Yoram Halevy, 2007. "Ellsberg Revisited: An Experimental Study," Econometrica, Econometric Society, vol. 75(2), pages 503-536, March.
    22. Peter Klibanoff & Massimo Marinacci & Sujoy Mukerji, 2005. "A Smooth Model of Decision Making under Ambiguity," Econometrica, Econometric Society, vol. 73(6), pages 1849-1892, November.
    23. repec:pri:cepsud:161blinder is not listed on IDEAS
    24. Ledyard, John & Hanson, Robin & Ishikida, Takashi, 2009. "An experimental test of combinatorial information markets," Journal of Economic Behavior & Organization, Elsevier, vol. 69(2), pages 182-189, February.
    25. Michael Abramowicz, 2007. "The Hidden Beauty of the Quadratic Market Scoring Rule: A Uniform Liquidity Market Maker, with Variations," Journal of Prediction Markets, University of Buckingham Press, vol. 1(2), pages 111-125, July.
    26. Olivier Armantier & Nicolas Treich, 2009. "Subjective Probabilities In Games: An Application To The Overbidding Puzzle," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 50(4), pages 1079-1102, November.
    27. Urs Fischbacher & Simon Gaechter, 2008. "Heterogeneous Social Preferences and the Dynamics of Free Riding in Public Good Experiments," TWI Research Paper Series 27, Thurgauer Wirtschaftsinstitut, Universität Konstanz.
    28. Mariana Blanco & Dirk Engelmann & Alexander Koch & Hans-Theo Normann, 2010. "Belief elicitation in experiments: is there a hedging problem?," Experimental Economics, Springer;Economic Science Association, vol. 13(4), pages 412-438, December.
    29. Stefan T. Trautmann & Gijs Kuilen, 2015. "Belief Elicitation: A Horse Race among Truth Serums," Economic Journal, Royal Economic Society, vol. 125(589), pages 2116-2135, December.
    30. Theo Offerman & Joep Sonnemans & Gijs Van De Kuilen & Peter P. Wakker, 2009. "A Truth Serum for Non-Bayesians: Correcting Proper Scoring Rules for Risk Attitudes ," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 76(4), pages 1461-1489.
    31. Robert W. Hahn & Paul Tetlock, 2006. "Information Markets: A New Way of Making Decisions," Books, American Enterprise Institute, number 51409, September.
    32. Charles F. Manski, 2004. "Measuring Expectations," Econometrica, Econometric Society, vol. 72(5), pages 1329-1376, September.
    33. Robin Hanson, 2003. "Combinatorial Information Market Design," Information Systems Frontiers, Springer, vol. 5(1), pages 107-119, January.
    34. Franklin Allen, 1987. "Notes--Discovering Personal Probabilities When Utility Functions are Unknown," Management Science, INFORMS, vol. 33(4), pages 542-544, April.
    35. Armantier, Olivier & Boly, Amadou, 2011. "A controlled field experiment on corruption," European Economic Review, Elsevier, vol. 55(8), pages 1072-1082.
    36. Manski, Charles F., 2002. "Identification of decision rules in experiments on simple games of proposal and response," European Economic Review, Elsevier, vol. 46(4-5), pages 880-891, May.
    37. Daniel Read, 2005. "Monetary incentives, what are they good for?," Journal of Economic Methodology, Taylor & Francis Journals, vol. 12(2), pages 265-276.
    38. Yaw Nyarko & Andrew Schotter, 2002. "An Experimental Study of Belief Learning Using Elicited Beliefs," Econometrica, Econometric Society, vol. 70(3), pages 971-1005, May.
    39. Drazen Prelec, 1998. "The Probability Weighting Function," Econometrica, Econometric Society, vol. 66(3), pages 497-528, May.
    40. Croson, Rachel T. A., 2000. "Thinking like a game theorist: factors affecting the frequency of equilibrium play," Journal of Economic Behavior & Organization, Elsevier, vol. 41(3), pages 299-314, March.
    41. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    42. McKelvey, Richard D & Page, Talbot, 1990. "Public and Private Information: An Experimental Study of Information Pooling," Econometrica, Econometric Society, vol. 58(6), pages 1321-1339, November.
    43. Karni, Edi, 1999. "Elicitation of Subjective Probabilities When Preferences Are State-Dependent," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 40(2), pages 479-486, May.
    Full references (including those not matched with items on IDEAS)

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    2. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    3. Charness, Gary & Gneezy, Uri & Rasocha, Vlastimil, 2021. "Experimental methods: Eliciting beliefs," Journal of Economic Behavior & Organization, Elsevier, vol. 189(C), pages 234-256.
    4. Eyting, Markus & Schmidt, Patrick, 2021. "Belief elicitation with multiple point predictions," European Economic Review, Elsevier, vol. 135(C).
    5. Steffen Andersen & John Fountain & Glenn Harrison & E. Rutström, 2014. "Estimating subjective probabilities," Journal of Risk and Uncertainty, Springer, vol. 48(3), pages 207-229, June.
    6. Mariana Blanco & Dirk Engelmann & Alexander Koch & Hans-Theo Normann, 2010. "Belief elicitation in experiments: is there a hedging problem?," Experimental Economics, Springer;Economic Science Association, vol. 13(4), pages 412-438, December.
    7. Tsakas, Elias, 2018. "Robust scoring rules," Research Memorandum 023, Maastricht University, Graduate School of Business and Economics (GSBE).
    8. Theo Offerman & Asa B. Palley, 2016. "Lossed in translation: an off-the-shelf method to recover probabilistic beliefs from loss-averse agents," Experimental Economics, Springer;Economic Science Association, vol. 19(1), pages 1-30, March.
    9. Folli, Dominik & Wolff, Irenaeus, 2022. "Biases in belief reports," Journal of Economic Psychology, Elsevier, vol. 88(C).
    10. Li Hao & Daniel Houser, 2012. "Belief elicitation in the presence of naïve respondents: An experimental study," Journal of Risk and Uncertainty, Springer, vol. 44(2), pages 161-180, April.
    11. Polonio, Luca & Coricelli, Giorgio, 2019. "Testing the level of consistency between choices and beliefs in games using eye-tracking," Games and Economic Behavior, Elsevier, vol. 113(C), pages 566-586.
    12. Costa-Gomes, Miguel A. & Huck, Steffen & Weizsäcker, Georg, 2014. "Beliefs and actions in the trust game: Creating instrumental variables to estimate the causal effect," EconStor Open Access Articles and Book Chapters, ZBW - Leibniz Information Centre for Economics, vol. 88, pages 298-309.
    13. Claudia Neri, 2015. "Eliciting beliefs in continuous-choice games: a double auction experiment," Experimental Economics, Springer;Economic Science Association, vol. 18(4), pages 569-608, December.
    14. Stefan T. Trautmann & Gijs Kuilen, 2015. "Belief Elicitation: A Horse Race among Truth Serums," Economic Journal, Royal Economic Society, vol. 125(589), pages 2116-2135, December.
    15. Florian Artinger & Filippos Exadaktylos & Hannes Koppel & Lauri Sääksvuori, 2010. "Applying Quadratic Scoring Rule transparently in multiple choice settings: A note," ThE Papers 10/01, Department of Economic Theory and Economic History of the University of Granada..
    16. Chen Li & Uyanga Turmunkh & Peter P. Wakker, 2019. "Trust as a decision under ambiguity," Experimental Economics, Springer;Economic Science Association, vol. 22(1), pages 51-75, March.
    17. Simon Gächter & Elke Renner, 2010. "The effects of (incentivized) belief elicitation in public goods experiments," Experimental Economics, Springer;Economic Science Association, vol. 13(3), pages 364-377, September.
    18. Simon Gächter & Elke Renner, 2010. "The effects of (incentivized) belief elicitation in public goods experiments," Experimental Economics, Springer;Economic Science Association, vol. 13(3), pages 364-377, September.
    19. Blanco, Mariana & Engelmann, Dirk & Koch, Alexander K. & Normann, Hans-Theo, 2014. "Preferences and beliefs in a sequential social dilemma: a within-subjects analysis," Games and Economic Behavior, Elsevier, vol. 87(C), pages 122-135.
    20. Harrison, Glenn W. & Martínez-Correa, Jimmy & Swarthout, J. Todd, 2014. "Eliciting subjective probabilities with binary lotteries," Journal of Economic Behavior & Organization, Elsevier, vol. 101(C), pages 128-140.

    More about this item

    Keywords

    Belief elicitation; Scoring rules; Experimental economics;
    All these keywords.

    JEL classification:

    • D81 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Criteria for Decision-Making under Risk and Uncertainty
    • D84 - Microeconomics - - Information, Knowledge, and Uncertainty - - - Expectations; Speculations
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:eee:eecrev:v:62:y:2013:i:c:p:17-40. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Catherine Liu (email available below). General contact details of provider: http://www.elsevier.com/locate/eer .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.