IDEAS home Printed from https://ideas.repec.org/p/jgu/wpaper/1818.html
   My bibliography  Save this paper

Belief Elicitation with Multiple Point Predictions

Author

Listed:
  • Markus Eyting

    (Johannes Gutenberg-University)

  • Patrick Schmidt

    (HITS gGmbH Heidelberg)

Abstract

We propose a simple, incentive compatible procedure based on binarized linear scoring rules to elicit beliefs about real-valued outcomes - multiple point predictions. Simultaneously eliciting multiple point predictions with linear incentives reveals the subjective probability distribution without pre-defined intervals or probabilistic statements. We show that the approach is theoretically as robust as existing methods, while adapting flexibly to different beliefs. In a laboratory experiment, we compare our procedure to the standard approach of eliciting discrete probabilities on pre-defined intervals. We find that elicitation with multiple point predictions is faster, perceived as less difficult and more consistent with a subsequent decision. We further find that multiple point predictions are more accurate if beliefs vary between participants. Finally, we provide experimental evidence that pre-defined intervals anchor reports.

Suggested Citation

  • Markus Eyting & Patrick Schmidt, 2019. "Belief Elicitation with Multiple Point Predictions," Working Papers 1818, Gutenberg School of Management and Economics, Johannes Gutenberg-Universität Mainz, revised 16 Nov 2020.
  • Handle: RePEc:jgu:wpaper:1818
    as

    Download full text from publisher

    File URL: https://download.uni-mainz.de/RePEc/pdf/Discussion_Paper_1818.pdf
    File Function: First version, 2018
    Download Restriction: no
    ---><---

    References listed on IDEAS

    as
    1. Giordani, Paolo & Soderlind, Paul, 2003. "Inflation forecast uncertainty," European Economic Review, Elsevier, vol. 47(6), pages 1037-1059, December.
    2. Qu, Xiangyu, 2012. "A mechanism for eliciting a probability distribution," Economics Letters, Elsevier, vol. 115(3), pages 399-400.
    3. Charness, Gary & Dufwenberg, Martin, 2003. "Promises & Partnership," Research Papers in Economics 2003:3, Stockholm University, Department of Economics.
    4. Gneiting, Tilmann, 2011. "Making and Evaluating Point Forecasts," Journal of the American Statistical Association, American Statistical Association, vol. 106(494), pages 746-762.
    5. Engelberg, Joseph & Manski, Charles F. & Williams, Jared, 2009. "Comparing the Point Predictions and Subjective Probability Distributions of Professional Forecasters," Journal of Business & Economic Statistics, American Statistical Association, vol. 27, pages 30-41.
    6. James E. Matheson & Robert L. Winkler, 1976. "Scoring Rules for Continuous Probability Distributions," Management Science, INFORMS, vol. 22(10), pages 1087-1096, June.
    7. Michael Hurd & Maarten Van Rooij & Joachim Winter, 2011. "Stock market expectations of Dutch households," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 26(3), pages 416-436, April.
    8. Theo Offerman & Joep Sonnemans & Gijs Van De Kuilen & Peter P. Wakker, 2009. "A Truth Serum for Non-Bayesians: Correcting Proper Scoring Rules for Risk Attitudes ," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 76(4), pages 1461-1489.
    9. Johanna F. Ziegel, 2016. "Coherence And Elicitability," Mathematical Finance, Wiley Blackwell, vol. 26(4), pages 901-918, October.
    10. Harrison, Glenn W. & Martínez-Correa, Jimmy & Swarthout, J. Todd, 2014. "Eliciting subjective probabilities with binary lotteries," Journal of Economic Behavior & Organization, Elsevier, vol. 101(C), pages 128-140.
    11. Andrew Schotter & Isabel Trevino, 2014. "Belief Elicitation in the Laboratory," Annual Review of Economics, Annual Reviews, vol. 6(1), pages 103-128, August.
    12. Lahiri, Kajal & Teigland, Christie & Zaporowski, Mark, 1988. "Interest Rates and the Subjective Probability Distribution of Inflation Forecasts," Journal of Money, Credit and Banking, Blackwell Publishing, vol. 20(2), pages 233-248, May.
    13. Claudia Neri, 2015. "Eliciting beliefs in continuous-choice games: a double auction experiment," Experimental Economics, Springer;Economic Science Association, vol. 18(4), pages 569-608, December.
    14. Edi Karni, 2009. "A Mechanism for Eliciting Probabilities," Econometrica, Econometric Society, vol. 77(2), pages 603-606, March.
    15. Craig R. Fox & Robert T. Clemen, 2005. "Subjective Probability Assessment in Decision Analysis: Partition Dependence and Bias Toward the Ignorance Prior," Management Science, INFORMS, vol. 51(9), pages 1417-1432, September.
    16. Tanjim Hossain & Ryo Okui, 2013. "The Binarized Scoring Rule," The Review of Economic Studies, Review of Economic Studies Ltd, vol. 80(3), pages 984-1001.
    17. Fabian Gouret & Guillaume Hollard, 2011. "When Kahneman meets Manski: Using dual systems of reasoning to interpret subjective expectations of equity returns," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 26(3), pages 371-392, April.
    18. Katja Kaufmann & Luigi Pistaferri, 2009. "Disentangling Insurance and Information in Intertemporal Consumption Choices," American Economic Review, American Economic Association, vol. 99(2), pages 387-392, May.
    19. Orazio Attanasio & Britta Augsburg, 2016. "Subjective Expectations and Income Processes in Rural India," Economica, London School of Economics and Political Science, vol. 83(331), pages 416-442, July.
    20. Charles F. Manski, 2004. "Measuring Expectations," Econometrica, Econometric Society, vol. 72(5), pages 1329-1376, September.
    21. Diebold, Francis X & Gunther, Todd A & Tay, Anthony S, 1998. "Evaluating Density Forecasts with Applications to Financial Risk Management," International Economic Review, Department of Economics, University of Pennsylvania and Osaka University Institute of Social and Economic Research Association, vol. 39(4), pages 863-883, November.
    22. Demuynck, Thomas, 2013. "A mechanism for eliciting the mean and quantiles of a random variable," Economics Letters, Elsevier, vol. 121(1), pages 121-123.
    23. Kristin J. Kleinjans & Arthur Van Soest, 2014. "Rounding, Focal Point Answers And Nonresponse To Subjective Probability Questions," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 29(4), pages 567-585, June.
    24. repec:cup:judgdm:v:5:y:2010:i:7:p:467-476 is not listed on IDEAS
    25. Gary Charness & Martin Dufwenberg, 2006. "Promises and Partnership," Econometrica, Econometric Society, vol. 74(6), pages 1579-1601, November.
    26. Palfrey, Thomas R. & Wang, Stephanie W., 2009. "On eliciting beliefs in strategic games," Journal of Economic Behavior & Organization, Elsevier, vol. 71(2), pages 98-109, August.
    27. Fabio Bellini & Valeria Bignozzi, 2015. "On elicitable risk measures," Quantitative Finance, Taylor & Francis Journals, vol. 15(5), pages 725-733, May.
    28. Dominitz, Jeff, 2001. "Estimation of income expectations models using expectations and realization data," Journal of Econometrics, Elsevier, vol. 102(2), pages 165-195, June.
    29. McKenzie, David & Gibson, John & Stillman, Steven, 2013. "A land of milk and honey with streets paved with gold: Do emigrants have over-optimistic expectations about incomes abroad?," Journal of Development Economics, Elsevier, vol. 102(C), pages 116-127.
    30. Jay Lu, 2016. "Random Choice and Private Information," Econometrica, Econometric Society, vol. 84, pages 1983-2027, November.
    31. Chen, Daniel L. & Schonger, Martin & Wickens, Chris, 2016. "oTree—An open-source platform for laboratory, online, and field experiments," Journal of Behavioral and Experimental Finance, Elsevier, vol. 9(C), pages 88-97.
    32. Adeline Delavande & Xavier Giné & David McKenzie, 2011. "Eliciting probabilistic expectations with visual aids in developing countries: how sensitive are answers to variations in elicitation design?," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 26(3), pages 479-497, April.
    33. Dufwenberg, Martin & Gneezy, Uri, 2000. "Measuring Beliefs in an Experimental Lost Wallet Game," Games and Economic Behavior, Elsevier, vol. 30(2), pages 163-182, February.
    34. Manski, Charles F. & Neri, Claudia, 2013. "First- and second-order subjective expectations in strategic decision-making: Experimental evidence," Games and Economic Behavior, Elsevier, vol. 81(C), pages 232-254.
    35. Ruth Vargas Hill, 2010. "Liberalisation and Producer Price Risk: Examining Subjective Expectations in the Ugandan Coffee Market," Journal of African Economies, Centre for the Study of African Economies, vol. 19(4), pages 433-458, August.
    36. L. Bissonnette & J. de Bresser, 2018. "Eliciting Subjective Survival Curves: Lessons from Partial Identification," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 36(3), pages 505-515, July.
    37. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    38. Efe A. Ok, 2007. "Preliminaries of Real Analysis, from Real Analysis with Economic Applications," Introductory Chapters, in: Real Analysis with Economic Applications, Princeton University Press.
    39. Yaw Nyarko & Andrew Schotter, 2002. "An Experimental Study of Belief Learning Using Elicited Beliefs," Econometrica, Econometric Society, vol. 70(3), pages 971-1005, May.
    40. Engle, Robert F. & White (the late), Halbert (ed.), 1999. "Cointegration, Causality, and Forecasting: Festschrift in Honour of Clive W. J. Granger," OUP Catalogue, Oxford University Press, number 9780198296836.
    41. Tilmann Gneiting & Fadoua Balabdaoui & Adrian E. Raftery, 2007. "Probabilistic forecasts, calibration and sharpness," Journal of the Royal Statistical Society Series B, Royal Statistical Society, vol. 69(2), pages 243-268, April.
    42. Adeline Delavande, 2008. "Measuring revisions to subjective expectations," Journal of Risk and Uncertainty, Springer, vol. 36(1), pages 43-82, February.
    43. Gneiting, Tilmann & Raftery, Adrian E., 2007. "Strictly Proper Scoring Rules, Prediction, and Estimation," Journal of the American Statistical Association, American Statistical Association, vol. 102, pages 359-378, March.
    44. Charles Bellemare & Luc Bissonnette & Sabine Kröger, 2012. "Flexible Approximation of Subjective Expectations Using Probability Questions," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(1), pages 125-131.
    45. McKelvey, Richard D & Page, Talbot, 1990. "Public and Private Information: An Experimental Study of Information Pooling," Econometrica, Econometric Society, vol. 58(6), pages 1321-1339, November.
    46. Delavande, Adeline & Giné, Xavier & McKenzie, David, 2011. "Measuring subjective expectations in developing countries: A critical review and new evidence," Journal of Development Economics, Elsevier, vol. 94(2), pages 151-163, March.
    47. Gianna Boero & Jeremy Smith & Kenneth F. Wallis, 2015. "The Measurement and Characteristics of Professional Forecasters' Uncertainty," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 30(7), pages 1029-1046, November.
    48. Charles Bellemare & Luc Bissonnette & Sabine Kröger, 2011. "Flexible Approximation of Subjective Expectations Using Probability Questions," Journal of Business & Economic Statistics, Taylor & Francis Journals, vol. 30(1), pages 125-131, April.
    49. Garthwaite, Paul H. & Kadane, Joseph B. & O'Hagan, Anthony, 2005. "Statistical Methods for Eliciting Probability Distributions," Journal of the American Statistical Association, American Statistical Association, vol. 100, pages 680-701, June.
    50. Lahiri, Kajal & Teigland, Christie, 1987. "On the normality of probability distributions of inflation and GNP forecasts," International Journal of Forecasting, Elsevier, vol. 3(2), pages 269-279.
    51. Harrison, Glenn W. & Martínez-Correa, Jimmy & Swarthout, J. Todd & Ulm, Eric R., 2017. "Scoring rules for subjective probability distributions," Journal of Economic Behavior & Organization, Elsevier, vol. 134(C), pages 430-448.
    52. Guiso, Luigi & Jappelli, Tullio & Pistaferri, Luigi, 2002. "An Empirical Analysis of Earnings and Employment Risk," Journal of Business & Economic Statistics, American Statistical Association, vol. 20(2), pages 241-253, April.
    53. Zarnowitz, Victor & Lambros, Louis A, 1987. "Consensus and Uncertainty in Economic Prediction," Journal of Political Economy, University of Chicago Press, vol. 95(3), pages 591-621, June.
    54. Hajo Holzmann & Matthias Eulert, 2014. "The role of the information set for forecasting - with applications to risk management," Papers 1404.7653, arXiv.org.
    55. Manski, Charles F. & Molinari, Francesca, 2010. "Rounding Probabilistic Expectations in Surveys," Journal of Business & Economic Statistics, American Statistical Association, vol. 28(2), pages 219-231.
    56. Brenner, Lyle A. & Koehler, Derek J. & Liberman, Varda & Tversky, Amos, 1996. "Overconfidence in Probability and Frequency Judgments: A Critical Examination," Organizational Behavior and Human Decision Processes, Elsevier, vol. 65(3), pages 212-219, March.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Patrick Schmidt, 2019. "Eliciting ambiguity with mixing bets," Papers 1902.07447, arXiv.org, revised Jul 2022.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Eyting, Markus & Schmidt, Patrick, 2021. "Belief elicitation with multiple point predictions," European Economic Review, Elsevier, vol. 135(C).
    2. Charness, Gary & Gneezy, Uri & Rasocha, Vlastimil, 2021. "Experimental methods: Eliciting beliefs," Journal of Economic Behavior & Organization, Elsevier, vol. 189(C), pages 234-256.
    3. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    4. Folli, Dominik & Wolff, Irenaeus, 2022. "Biases in belief reports," Journal of Economic Psychology, Elsevier, vol. 88(C).
    5. Steffen Andersen & John Fountain & Glenn Harrison & E. Rutström, 2014. "Estimating subjective probabilities," Journal of Risk and Uncertainty, Springer, vol. 48(3), pages 207-229, June.
    6. Dominik Bauer & Irenaeus Wolff, 2018. "Biases in Beliefs: Experimental Evidence," TWI Research Paper Series 109, Thurgauer Wirtschaftsinstitut, Universität Konstanz.
    7. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    8. Bauer, Dominik & Wolff, Irenaeus, 2019. "Biases in Beliefs," VfS Annual Conference 2019 (Leipzig): 30 Years after the Fall of the Berlin Wall - Democracy and Market Economy 203601, Verein für Socialpolitik / German Economic Association.
    9. Charles F. Manski, 2018. "Survey Measurement of Probabilistic Macroeconomic Expectations: Progress and Promise," NBER Macroeconomics Annual, University of Chicago Press, vol. 32(1), pages 411-471.
    10. Crosetto, Paolo & Filippin, Antonio & Katuščák, Peter & Smith, John, 2020. "Central tendency bias in belief elicitation," Journal of Economic Psychology, Elsevier, vol. 78(C).
    11. de Haan, Thomas, 2020. "Eliciting belief distributions using a random two-level partitioning of the state space," Working Papers in Economics 1/20, University of Bergen, Department of Economics.
    12. Tsakas, Elias, 2018. "Robust scoring rules," Research Memorandum 023, Maastricht University, Graduate School of Business and Economics (GSBE).
    13. Li Hao & Daniel Houser, 2012. "Belief elicitation in the presence of naïve respondents: An experimental study," Journal of Risk and Uncertainty, Springer, vol. 44(2), pages 161-180, April.
    14. Alexander Glas & Matthias Hartmann, 2022. "Uncertainty measures from partially rounded probabilistic forecast surveys," Quantitative Economics, Econometric Society, vol. 13(3), pages 979-1022, July.
    15. Armantier, Olivier & Treich, Nicolas, 2013. "Eliciting beliefs: Proper scoring rules, incentives, stakes and hedging," European Economic Review, Elsevier, vol. 62(C), pages 17-40.
    16. Polonio, Luca & Coricelli, Giorgio, 2019. "Testing the level of consistency between choices and beliefs in games using eye-tracking," Games and Economic Behavior, Elsevier, vol. 113(C), pages 566-586.
    17. Claudia Neri, 2015. "Eliciting beliefs in continuous-choice games: a double auction experiment," Experimental Economics, Springer;Economic Science Association, vol. 18(4), pages 569-608, December.
    18. Guillaume Hollard & Sébastien Massoni & Jean-Christophe Vergnaud, 2016. "In search of good probability assessors: an experimental comparison of elicitation rules for confidence judgments," Theory and Decision, Springer, vol. 80(3), pages 363-387, March.
    19. Clements, Michael P., 2018. "Are macroeconomic density forecasts informative?," International Journal of Forecasting, Elsevier, vol. 34(2), pages 181-198.
    20. Heiss, Florian & Hurd, Michael & van Rooij, Maarten & Rossmann, Tobias & Winter, Joachim, 2022. "Dynamics and heterogeneity of subjective stock market expectations," Journal of Econometrics, Elsevier, vol. 231(1), pages 213-231.

    More about this item

    Keywords

    elicitation of subjective expectations; partial identification; quantiles; overconfidence; experiment;
    All these keywords.

    NEP fields

    This paper has been announced in the following NEP Reports:

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:jgu:wpaper:1818. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Research Unit IPP (email available below). General contact details of provider: https://edirc.repec.org/data/vlmaide.html .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.