IDEAS home Printed from https://ideas.repec.org/a/cup/judgdm/v18y2023ip-_27.html
   My bibliography  Save this article

Comparing input interfaces to elicit belief distributions

Author

Listed:
  • Crosetto, Paolo
  • de Haan, Thomas

Abstract

This paper introduces a new software interface to elicit belief distributions of any shape: Click-and-Drag. The interface was tested against the state of the art in the experimental literature—a text-based interface and multiple sliders—and in the online forecasting industry—a distribution-manipulation interface similar to the one used by the most popular crowd-forecasting website. By means of a pre-registered experiment on Amazon Mechanical Turk, quantitative data on the accuracy of reported beliefs in a series of induced-value scenarios varying by granularity, shape, and time constraints, as well as subjective data on user experience were collected. Click-and-Drag outperformed all other interfaces by accuracy and speed, and was self-reported as being more intuitive and less frustrating, confirming the pre-registered hypothesis. Aside of the pre-registered results, Click-and-Drag generated the least drop-out rate from the task, and scored best in a sentiment analysis of an open-ended general question. Further, the interface was used to collect homegrown predictions on temperature in New York City in 2022 and 2042. Click-and-Drag elicited distributions were smoother with less idiosyncratic spikes. Free and open source, ready to use oTree, Qualtrics and Limesurvey plugins for Click-and-Drag, and all other tested interfaces are available at https://beliefelicitation.github.io/.

Suggested Citation

  • Crosetto, Paolo & de Haan, Thomas, 2023. "Comparing input interfaces to elicit belief distributions," Judgment and Decision Making, Cambridge University Press, vol. 18, pages 1-1, January.
    • Handle: RePEc:cup:judgdm:v:18:y:2023:i::p:-_27
    as

    Download full text from publisher

    File URL: https://www.cambridge.org/core/product/identifier/S1930297523000219/type/journal_article
    File Function: link to article abstract page
    Download Restriction: no
    ---><---

    Other versions of this item:

    References listed on IDEAS

    as
    1. repec:cup:judgdm:v:9:y:2014:i:1:p:1-14 is not listed on IDEAS
    2. Olivier Armantier & Scott Nelson & Giorgio Topa & Wilbert van der Klaauw & Basit Zafar, 2016. "The Price Is Right: Updating Inflation Expectations in a Randomized Price Information Experiment," The Review of Economics and Statistics, MIT Press, vol. 98(3), pages 503-523, July.
    3. Stefan T. Trautmann & Gijs Kuilen, 2015. "Belief Elicitation: A Horse Race among Truth Serums," Economic Journal, Royal Economic Society, vol. 125(589), pages 2116-2135, December.
    4. Kryvtsov, Oleksiy & Petersen, Luba, 2021. "Central bank communication that works: Lessons from lab experiments," Journal of Monetary Economics, Elsevier, vol. 117(C), pages 760-780.
    5. Andrew Schotter & Isabel Trevino, 2014. "Belief Elicitation in the Laboratory," Annual Review of Economics, Annual Reviews, vol. 6(1), pages 103-128, August.
    6. Kriti Jain & Kanchan Mukherjee & J. Neil Bearden & Anil Gaba, 2013. "Unpacking the Future: A Nudge Toward Wider Subjective Confidence Intervals," Management Science, INFORMS, vol. 59(9), pages 1970-1987, September.
    7. Lionel Page & Daniel G. Goldstein, 2016. "Subjective beliefs about the income distribution and preferences for redistribution," Social Choice and Welfare, Springer;The Society for Social Choice and Welfare, vol. 47(1), pages 25-61, June.
    8. Wändi Bruine De Bruin & Charles F. Manski & Giorgio Topa & Wilbert van der Klaauw, 2011. "Measuring consumer uncertainty about future inflation," Journal of Applied Econometrics, John Wiley & Sons, Ltd., vol. 26(3), pages 454-478, April.
    9. Charles F. Manski, 2004. "Measuring Expectations," Econometrica, Econometric Society, vol. 72(5), pages 1329-1376, September.
    10. David Danz & Lise Vesterlund & Alistair J. Wilson, 2022. "Belief Elicitation and Behavioral Incentive Compatibility," American Economic Review, American Economic Association, vol. 112(9), pages 2851-2883, September.
    11. Jeff Dominitz & Charles F. Manski, 2004. "How Should We Measure Consumer Confidence?," Journal of Economic Perspectives, American Economic Association, vol. 18(2), pages 51-66, Spring.
    12. Harrison, Glenn W. & Martínez-Correa, Jimmy & Swarthout, J. Todd & Ulm, Eric R., 2017. "Scoring rules for subjective probability distributions," Journal of Economic Behavior & Organization, Elsevier, vol. 134(C), pages 430-448.
    13. Rholes, Ryan & Petersen, Luba, 2021. "Should central banks communicate uncertainty in their projections?," Journal of Economic Behavior & Organization, Elsevier, vol. 183(C), pages 320-341.
    14. repec:cup:judgdm:v:10:y:2015:i:5:p:456-468 is not listed on IDEAS
    15. Leemann, Lucas & Stoetzer, Lukas F. & Traunmüller, Richard, 2021. "Eliciting Beliefs as Distributions in Online Surveys," Political Analysis, Cambridge University Press, vol. 29(4), pages 541-553, October.
    16. Glenn W. Harrison & Andre Hofmeyr & Harold Kincaid & Brian Monroe & Don Ross & Mark Schneider & J. Todd Swarthout, 2022. "Subjective beliefs and economic preferences during the COVID-19 pandemic," Experimental Economics, Springer;Economic Science Association, vol. 25(3), pages 795-823, June.
    17. Ernan Haruvy & Yaron Lahav & Charles N. Noussair, 2007. "Traders' Expectations in Asset Markets: Experimental Evidence," American Economic Review, American Economic Association, vol. 97(5), pages 1901-1920, December.
    18. Fairley, Kim & Parelman, Jacob M. & Jones, Matt & Carter, R. McKell, 2019. "Risky health choices and the Balloon Economic Risk Protocol," Journal of Economic Psychology, Elsevier, vol. 73(C), pages 15-33.
    19. Kröger, Sabine & Pierrot, Thibaud, 2019. "What point of a distribution summarises point predictions?," Discussion Papers, Research Unit: Market Behavior SP II 2019-212, WZB Berlin Social Science Center.
    Full references (including those not matched with items on IDEAS)

    Citations

    Citations are extracted by the CitEc Project, subscribe to its RSS feed for this item.
    as


    Cited by:

    1. Becker, Christoph & Duersch, Peter & Eife, Thomas, 2023. "Measuring Inflation Expectations: How the Response Scale Shapes Density Forecasts," Working Papers 0727, University of Heidelberg, Department of Economics.

    Most related items

    These are the items that most often cite the same works as this one and are cited by the same works as this one.
    1. Grewenig, Elisabeth & Lergetporer, Philipp & Werner, Katharina & Woessmann, Ludger, 2022. "Incentives, search engines, and the elicitation of subjective beliefs: Evidence from representative online survey experiments," Journal of Econometrics, Elsevier, vol. 231(1), pages 304-326.
    2. Crosetto, Paolo & Filippin, Antonio & Katuščák, Peter & Smith, John, 2020. "Central tendency bias in belief elicitation," Journal of Economic Psychology, Elsevier, vol. 78(C).
    3. Charness, Gary & Gneezy, Uri & Rasocha, Vlastimil, 2021. "Experimental methods: Eliciting beliefs," Journal of Economic Behavior & Organization, Elsevier, vol. 189(C), pages 234-256.
    4. Karl Schlag & James Tremewan & Joël Weele, 2015. "A penny for your thoughts: a survey of methods for eliciting beliefs," Experimental Economics, Springer;Economic Science Association, vol. 18(3), pages 457-490, September.
    5. Eyting, Markus & Schmidt, Patrick, 2021. "Belief elicitation with multiple point predictions," European Economic Review, Elsevier, vol. 135(C).
    6. Crump, Richard K. & Eusepi, Stefano & Tambalotti, Andrea & Topa, Giorgio, 2022. "Subjective intertemporal substitution," Journal of Monetary Economics, Elsevier, vol. 126(C), pages 118-133.
    7. Evans, George W. & Hommes, Cars & McGough, Bruce & Salle, Isabelle, 2022. "Are long-horizon expectations (de-)stabilizing? Theory and experiments," Journal of Monetary Economics, Elsevier, vol. 132(C), pages 44-63.
    8. Abe, Naohito & Ueno, Yuko, 2016. "The Mechanism of Inflation Expectation Formation among Consumers," RCESR Discussion Paper Series DP16-1, Research Center for Economic and Social Risks, Institute of Economic Research, Hitotsubashi University.
    9. Polonio, Luca & Coricelli, Giorgio, 2019. "Testing the level of consistency between choices and beliefs in games using eye-tracking," Games and Economic Behavior, Elsevier, vol. 113(C), pages 566-586.
    10. Kai Barron, 2021. "Belief updating: does the ‘good-news, bad-news’ asymmetry extend to purely financial domains?," Experimental Economics, Springer;Economic Science Association, vol. 24(1), pages 31-58, March.
    11. Charles F. Manski, 2018. "Survey Measurement of Probabilistic Macroeconomic Expectations: Progress and Promise," NBER Macroeconomics Annual, University of Chicago Press, vol. 32(1), pages 411-471.
    12. Tom Lane, 2023. "The strategic use of social identity," Discussion Papers 2023-01, The Centre for Decision Research and Experimental Economics, School of Economics, University of Nottingham.
    13. Bańnkowska, Katarzyna & Borlescu, Ana Maria & Charalambakis, Evangelos & Da Silva, António Dias & Di Laurea, Davide & Dossche, Maarten & Georgarakos, Dimitris & Honkkila, Juha & Kennedy, Neale & Kenny, 2021. "ECB Consumer Expectations Survey: an overview and first evaluation," Occasional Paper Series 287, European Central Bank.
    14. Armantier, Olivier & Koşar, Gizem & Pomerantz, Rachel & Skandalis, Daphné & Smith, Kyle & Topa, Giorgio & van der Klaauw, Wilbert, 2021. "How economic crises affect inflation beliefs: Evidence from the Covid-19 pandemic," Journal of Economic Behavior & Organization, Elsevier, vol. 189(C), pages 443-469.
    15. Burdea, Valeria & Woon, Jonathan, 2022. "Online belief elicitation methods," Journal of Economic Psychology, Elsevier, vol. 90(C).
    16. Markus Eyting & Patrick Schmidt, 2019. "Belief Elicitation with Multiple Point Predictions," Working Papers 1818, Gutenberg School of Management and Economics, Johannes Gutenberg-Universität Mainz, revised 16 Nov 2020.
    17. Naohito Abe & Yuko Ueno, 2016. "The Mechanism of Inflation Expectation Formation among Consumers," UTokyo Price Project Working Paper Series 064, University of Tokyo, Graduate School of Economics.
    18. Lata Gangadharan & Philip J. Grossman & Nina Xue, 2022. "Stepping Stone: Identifying self-image concerns from motivated beliefs: Does it matter how and whom you ask?," Monash Economics Working Papers 2022-05, Monash University, Department of Economics.
    19. Niu, G., 2014. "Essays on subjective expectations and mortality trends," Other publications TiSEM b9f72836-d8ad-478b-adca-4, Tilburg University, School of Economics and Management.
    20. Petersen, Luba & Rholes, Ryan, 2022. "Macroeconomic expectations, central bank communication, and background uncertainty: A COVID-19 laboratory experiment," Journal of Economic Dynamics and Control, Elsevier, vol. 143(C).

    More about this item

    JEL classification:

    • C88 - Mathematical and Quantitative Methods - - Data Collection and Data Estimation Methodology; Computer Programs - - - Other Computer Software
    • C91 - Mathematical and Quantitative Methods - - Design of Experiments - - - Laboratory, Individual Behavior

    Statistics

    Access and download statistics

    Corrections

    All material on this site has been provided by the respective publishers and authors. You can help correct errors and omissions. When requesting a correction, please mention this item's handle: RePEc:cup:judgdm:v:18:y:2023:i::p:-_27. See general information about how to correct material in RePEc.

    If you have authored this item and are not yet registered with RePEc, we encourage you to do it here. This allows to link your profile to this item. It also allows you to accept potential citations to this item that we are uncertain about.

    If CitEc recognized a bibliographic reference but did not link an item in RePEc to it, you can help with this form .

    If you know of missing items citing this one, you can help us creating those links by adding the relevant references in the same way as above, for each refering item. If you are a registered author of this item, you may also want to check the "citations" tab in your RePEc Author Service profile, as there may be some citations waiting for confirmation.

    For technical questions regarding this item, or to correct its authors, title, abstract, bibliographic or download information, contact: Kirk Stebbing (email available below). General contact details of provider: https://www.cambridge.org/jdm .

    Please note that corrections may take a couple of weeks to filter through the various RePEc services.

    IDEAS is a RePEc service. RePEc uses bibliographic data supplied by the respective publishers.