Fat Tails in Human Judgment: Empirical Evidence and Implications for the Aggregation of Estimates and Forecasts

How frequent are large disagreements in human judgment? The substantial literature relating to expert assessments of real-valued quantities and their aggregation almost universally assumes that judgment errors follow a jointly normal distribution. We investigate this question empirically using 73 data sets from four different sources that include over 169,000 estimates and forecasts. We find incontrovertible evidence for excess kurtosis: that is, of fat tails. Despite the diversity of the analyzed data, varying in how much uncertainty there is in the quantity being assessed and varying in the level of expertise and of sophistication of those making the assessments, we find surprising consistency in the frequency with which an individual judgment is in large disagreement with the consensus. Fitting a generalized normal distribution to the data, we find most estimates for the shape parameter to be between 0.9 and 1.6 (where 1 is the double-exponential distribution and 2 is the normal distribution). This has important implications, in particular for the aggregation of expert estimates and forecasts and for the construction of confidence intervals. We describe optimal Bayesian aggregation with fat tails and propose a simple average-median average heuristic that performs well for the range of empirically observed distributions.