An expert judgment model to predict early stages of the COVID-19 pandemic in the United States

From February to May 2020, experts in the modeling of infectious disease provided quantitative predictions and estimates of trends in the emerging COVID-19 pandemic in a series of 13 surveys. Data on existing transmission patterns were sparse when the pandemic began, but experts synthesized information available to them to provide quantitative, judgment-based assessments of the current and future state of the pandemic. We aggregated expert predictions into a single “linear pool” by taking an equally weighted average of their probabilistic statements. At a time when few computational models made public estimates or predictions about the pandemic, expert judgment provided (a) falsifiable predictions of short- and long-term pandemic outcomes related to reported COVID-19 cases, hospitalizations, and deaths, (b) estimates of latent viral transmission, and (c) counterfactual assessments of pandemic trajectories under different scenarios. The linear pool approach of aggregating expert predictions provided more consistently accurate predictions than any individual expert, although the predictive accuracy of a linear pool rarely provided the most accurate prediction. This work highlights the importance that an expert linear pool could play in flexibly assessing a wide array of risks early in future emerging outbreaks, especially in settings where available data cannot yet support data-driven computational modeling.Author summary: We asked experts in the modeling of infectious disease to submit probabilistic predictions of the spread and burden of SARS-CoV-2/COVID-19 from February to May, 2020 in an effort to support public health decision making. Expert predictions were aggregated into a linear pool. We found experts could produce short and long term predictions related to the pandemic that could be compared to ground truth such as the number of cases occurring by the end of the week and predictions of unmeasurable outcomes such as latent viral transmission. Experts were also able to make counter factual predictions—predictions of an outcome assuming an action will continue or not continue. In addition, predictions built by aggregating individual expert predictions were less variable when compared to predictions made by individuals. Our work highlights that an expert linear pool is a fast, flexible tool that can support situational awareness for public health officials during an emerging outbreak.