Comparing Models for Early Warning Systems of Neglected Tropical Diseases

Background: Early warning systems (EWS) are management tools to predict the occurrence of epidemics of infectious diseases. While climate-based EWS have been developed for malaria, no standard protocol to evaluate and compare EWS has been proposed. Additionally, there are several neglected tropical diseases whose transmission is sensitive to environmental conditions, for which no EWS have been proposed, though they represent a large burden for the affected populations. Methodology/Principal Findings: In the present paper, an overview of the available linear and non-linear tools to predict seasonal time series of diseases is presented. Also, a general methodology to compare and evaluate models for prediction is presented and illustrated using American cutaneous leishmaniasis, a neglected tropical disease, as an example. The comparison of the different models using the predictive R2 for forecasts of “out-of-fit” data (data that has not been used to fit the models) shows that for the several linear and non-linear models tested, the best results were obtained for seasonal autoregressive (SAR) models that incorporate climatic covariates. An additional bootstrapping experiment shows that the relationship of the disease time series with the climatic covariates is strong and consistent for the SAR modeling approach. While the autoregressive part of the model is not significant, the exogenous forcing due to climate is always statistically significant. Prediction accuracy can vary from 50% to over 80% for disease burden at time scales of one year or shorter. Conclusions/Significance: This study illustrates a protocol for the development of EWS that includes three main steps: (i) the fitting of different models using several methodologies, (ii) the comparison of models based on the predictability of “out-of-fit” data, and (iii) the assessment of the robustness of the relationship between the disease and the variables in the model selected as best with an objective criterion. Author Summary: Early Warning Systems (EWS) are management tools to predict the occurrence of epidemics. They are based on the dependence of a given infectious disease on environmental variables. Although several neglected tropical diseases are sensitive to the effect of climate, our ability to predict their dynamics has been barely studied. In this paper, we use several models to determine if the relationship between cases and climatic variability is robust—that is, not simply an artifact of model choice. We propose that EWS should be based on results from several models that are to be compared in terms of their ability to predict future number of cases. We use a specific metric for this comparison known as the predictive R2, which measures the accuracy of the predictions. For example, an R2 of 1 indicates perfect accuracy for predictions that perfectly match observed cases. For cutaneous leishmaniasis, R2 values range from 72% to77%, well above predictions using mean seasonal values (64%). We emphasize that predictability should be evaluated with observations that have not been used to fit the model. Finally, we argue that EWS should incorporate climatic variables that are known to have a consistent relationship with the number of observed cases.