Percentage and Relative Error Measures in Forecast Evaluation

Properties of two large families of scale-free forecast accuracy measures that include popular measures such as mean absolute percentage error, relative error, and squared percentage error, are examined in this paper. We describe the optimal reports when forecasts are evaluated using these measures. We also provide analytic expressions for the optimal Bayes’ act associated with these measures under a general power transformation for several well-known probability distributions. We then show that using measures from these two families may inadvertently provide incentives for either pessimism or optimism among forecasters, i.e., rewarding underforecasts or overforecasts relative to some reference measure of central tendency. As an illustration of these concepts, we examine the use of these measures for model selection in a forecast aggregation example using stock price forecasts derived from the Thomson Reuters Institutional Brokers’ Estimate System. This example illustrates how aggregation methods that always yield lower estimates relative to the mean or median generally exhibit better scores using percentage error-based measures, while those that yield higher estimates compared to the mean or median will effectively rank higher when relative error-based measures are used.