Comparing input interfaces to elicit belief distributions

We develop an intuitive, Click-and-Drag interface to elicit continuous belief distributions of any shape. We test this interface against the state of the art in the experimental literature – a text-based interface and multiple sliders – and in the online forecasting industry – a distribution-manipulation interface similar to the one used at Metaculus, a crowd-forecasting website. By means of a pre-registered experiment on Amazon Mechanical Turk we collect quantitative data on the convergence speed and accuracy of reported beliefs in a series of induced-value scenarios varying by granularity, shape, and time constraints. We also collect subjective data on ease of use, frustration and understanding. Results show that the click-and-drag interface outperforms all others by accuracy and speed, and is self-reported as being more intuitive and less frustrating than other interfaces, confirming our pre-registered hypothesis. Besides pre-registration, we report that the click-and-drag interface generates the least drop-out rate from the task, and scores best in a sentiment analysis of an open-ended general question. Further, we use the interfaces to collect homegrown beliefs on temperature in New York City in 2022 and 2042. On average, all subjects overshoot the real temperature for 2022 by about 2°F, and all anticipate further global warming in the order of 2.3°F; these forecasts are by and large not impacted by the interface used to elicit them. We provide a free and open source, ready to use oTree and Qualtrics plugin of our click-and-drag and all other tested interfaces available at https://beliefelicitation.github.io/.