Optimal out‐of‐sample forecast evaluation under stationarity

It is a common practice to split a time series into an in‐sample and pseudo‐out‐of‐sample segments and estimate the out‐of‐sample loss for a given statistical model by evaluating forecasting performance over the pseudo‐out‐of‐sample segment. We propose an alternative estimator of the out‐of‐sample loss, which, contrary to the conventional wisdom, utilizes criteria measured both in‐ and out‐of‐sample via a carefully constructed system of affine weights. We prove that, provided that the time series is stationary, the proposed estimator is the best linear unbiased estimator of the out‐of‐sample loss and outperforms the conventional estimator in terms of sampling variability. Application of the optimal estimator to Diebold–Mariano type tests of predictive ability leads to a substantial power gain without increasing finite sample size distortions. An extensive evaluation on real‐world time series from the M4 forecasting competition confirms superiority of the proposed estimator and also demonstrates substantial robustness to violations of the underlying assumption of stationarity.