The influence of ensemble size and composition on the performance of combined real-time COVID-19 forecasts

During infectious disease outbreaks, short-term forecasts can play an important role for both decision makers and the general public. While previous research has shown that combining individual forecasts into an ensemble improves accuracy and consistency, practical guidance for organisers of multi-model prediction platforms on how to construct an ensemble has been scarce. In particular, it is not entirely clear how ensemble performance relates to the size of the underlying model pool, a relevant question when relying on voluntary contributions from modelling teams that face competing priorities. Furthermore, the exact composition of an ensemble forecast may influence its performance. Ensembles can either include all models equally or, alternatively, discriminate based on past performance or other characteristics. Using data from the European COVID-19 Forecast Hub we investigated these questions, with the aim of offering practical guidance to organisers of multi-model prediction platforms during infectious disease outbreaks. We found that including more models both improved and stabilised aggregate ensemble performance, while selecting for better component models did not yield any particular advantage. Diversity among models, whether measured numerically or qualitatively, did not have a clear impact on ensemble performance. These results suggest that for those soliciting contributions to collaborative ensembles there are more obvious gains to be made from increasing participation to moderate levels than from optimising component models.Author summary: Infectious disease outbreaks such as the recent COVID-19 pandemic present uncertain situations for governments, health agencies and the general public. Short-term forecasts of key indicators, such as case numbers and deaths from the disease, can help inform expectations about how an outbreak is likely to develop in the near future. These forecasts are often produced using computer models that differ in their underlying assumptions and use of available data. One common approach to improve prediction quality is to collect forecasts from several such models and combine them into a so-called ensemble forecast. However, for organisers of these forecasting efforts it is often unclear how to best construct such an ensemble, in particular how many individual forecasts to collect or whether some models’ forecasts should be selected over others. Using forecasts submitted to the European COVID-19 Forecast Hub, we examined how ensemble performance depends on the number and selection of contributing model forecasts. We found that ensembles generally became more accurate and more stable as additional models’ forecasts were included. In contrast, selectively including only models with better past performance did not consistently improve results. We also found no clear evidence that greater diversity among modelling approaches led to better ensemble forecasts. Our findings suggest that ensemble forecasting efforts benefit more from encouraging broad participation to obtain a moderate number of forecasts than from selecting individual models’ forecasts.