Quantile Probability Predictions: A Demonstrative Performance Analysis of Forecasts of US COVID-19 Deaths

An analytical framework is presented for the evaluation of quantile probability forecasts. It is demonstrated using weekly quantile forecasts of changes in the number of US COVID-19 deaths. Empirical quantiles are derived using the assumption that daily changes in a variable follow a normal distribution with time varying means and standard deviations, which can be assumed constant over short horizons such as one week. These empirical quantiles are used to evaluate quantile forecasts using the Mean Squared Quantile Score (MSQS), which, in turn, is decomposed into sub-components involving bias, resolution and error variation to identify specific aspects of performance, which highlight the strengths and weaknesses of forecasts. The framework is then extended to test if performance enhancement can be achieved by combining diverse forecasts from different sources. The demonstration illustrates that the technique can effectively evaluate quantile forecasting performance based on a limited number of data points, which is crucial in emergency situations such as forecasting pandemic behavior. It also shows that combining the predictions with quantile probability forecasts generated from an Autoregressive Order One, AR(1) model provided substantially improved performance. The implications of these findings are discussed, suggestions are offered for future research and potential limitations are considered.