Social structure, endogenous diversity, and collective accuracy

Markets, democracies, and organizations rely on accurate aggregate predictions to function properly. A large literature explains how accuracy can arise from diverse predictive models, typically captured as independent or non-perfectly correlated signals. Yet, that literature largely ignores how the diversity of models arises and is maintained. In this paper, we derive equilibrium levels of model diversity as a function of social structure, population size, the probability of experimentation, and the number of available models by building on a theoretical framework used to study biodiversity. We then link model diversity to collective accuracy by generalizing the bias-variance decomposition formula. Assuming equally accurate models, we find that for large populations collective accuracy depends primarily on the diversity of available models and that for small populations, social structure and rates of experimentation also matter. We then show, contrary to intuition, that dividing a population into isolated sub groups does little to increase equilibrium diversity levels. We also extend the model to allow for heterogeneity in accuracy and selection effects.