The case for quantile forecasts: Enhancing reliability in the identification of forecaster talent

What is the best way to elicit forecasts if the goal is to measure the skill of forecasters? For continuous outcomes, there are two common approaches: elicit probabilities at fixed quantiles of the outcome distribution (e.g., what is the probability gas will cost less than $4.00 per gallon in 1 month?), or elicit quantiles at fixed probabilities (e.g., what is the price such that there is a 25% chance a gallon of gas will cost less than it in 1 month?). We compare these methods using a simulation study and a longitudinal survey in which 1,194 participants provided five waves of forecasts in both formats on a total of 36 real-world questions. Although probability elicitation is more ubiquitous in the literature, we show in our simulation and survey data that quantile elicitation allows us to estimate forecaster skill more efficiently with less measurement error. While quantile elicitation is time-consuming and associated with comprehension difficulty indicated by incoherent forecasts (violations of monotonicity), there is a clear reduction in incoherence across survey waves, suggesting comprehension of the quantile elicitation format improves with practice and is trainable.