Validation framework for epidemiological models with application to COVID-19 models

Mathematical models have been an important tool during the COVID-19 pandemic, for example to predict demand of critical resources such as medical devices, personal protective equipment and diagnostic tests. Many COVID-19 models have been developed. However, there is relatively little information available regarding reliability of model predictions. Here we present a general model validation framework for epidemiological models focused around predictive capability for questions relevant to decision-making end-users. COVID-19 models are typically comprised of multiple releases, and provide predictions for multiple localities, and these characteristics are systematically accounted for in the framework, which is based around a set of validation scores or metrics that quantify model accuracy of specific quantities of interest including: date of peak, magnitude of peak, rate of recovery, and monthly cumulative counts. We applied the framework to retrospectively assess accuracy of death predictions for four COVID-19 models, and accuracy of hospitalization predictions for one COVID-19 model (models for which sufficient data was publicly available). When predicting date of peak deaths, the most accurate model had errors of approximately 15 days or less, for releases 3-6 weeks in advance of the peak. Death peak magnitude relative errors were generally in the 50% range 3-6 weeks before peak. Hospitalization predictions were less accurate than death predictions. All models were highly variable in predictive accuracy across regions. Overall, our framework provides a wealth of information on the predictive accuracy of epidemiological models and could be used in future epidemics to evaluate new models or support existing modeling methodologies, and thereby aid in informed model-based public health decision making. The code for the validation framework is available at https://doi.org/10.5281/zenodo.7102854.Author summary: During the COVID-19 pandemic many mathematical models have been developed. These mathematical models provide forecasts of key quantities such as number of infectious cases, number of hospitalizations, and number of deaths, in the upcoming weeks in the locality of interest. However, the reliability of model predictions is unclear. Currently, there have been few techniques employed to validate the performance of COVID-19 models that have focused on quantities that are especially of interest to end-users, such as when deaths will peak or the magnitude of the peak. Here, we provide an epidemiological model validation framework focused on questions relevant to decision-makers that utilize COVID-19 model predictions. We analyze four COVID-19 models with our framework, examining the accuracy of each model in predicting the date and magnitude of a peak, recovery rate, and monthly cumulative counts, for predictions of deaths and of hospitalizations. Our results show that the mathematical models produce highly variable predictions across regions. Our framework demonstrates the need for predictive reliability of epidemiological models.