Evaluation of Forecasts

Evaluation of forecasts encompasses the processes of assessing both forecast quality and forecast value. These processes necessarily play key roles in any effort to improve forecasting performance or to enhance the usefulness of forecasts. A framework for forecast verification (the process of assessing forecast quality) based on the joint distribution of forecasts and observations — and on the conditional and marginal distributions derived from factorizations of this joint distribution — is described. The joint, conditional, and marginal distributions relate directly to basic aspects of forecast quality, and evaluation methods based on these distributions — and associated statistics and measures — provide a coherent, diagnostic approach to forecast verification. This approach — and its attendant methodology — is illustrated using a sample of probabilistic long-range weather forecasts. A decision-analytic approach to the problem of assessing the value of forecasts is outlined, and this approach is illustrated by considering the so-called fallowing-planting problem. In addition to providing estimates of the value of state-of-the-art and hypothetically improved long-range weather forecasts, the results of this case study illustrate some of the fundamental properties of quality/value relationships. These properties include the inherent nonlinearity of such relationships and the existence of quality thresholds below which the forecasts are of no value. The sufficiency relation is used to explore quality/value relationships; this relation embodies the conditions that must exist between the joint distributions of two forecasting systems to ensure that one system’s forecasts are better in all respects (i.e., in terms of quality and value) than the other system’s forecasts. The applicability of the sufficiency relation is illustrated by comparing forecasting systems that produce prototypical long-range weather forecasts. This application also demonstrates that quality/value reversals can occur when the multifaceted nature of forecast quality is not respected. Some outstanding problems in forecast evaluation are identified and briefly discussed. Recommendations are made regarding improvements in evaluation methods and practices.