Dynamical errors in machine learning forecasts

Error metrics such as mean squared error (MSE) are widely used to evaluate machine learning (ML) forecasts. However, these metrics provide little information on whether the forecasts preserve the dynamical properties of the underlying system – a critical yet often overlooked question in many scientific and engineering applications. This work investigates how standard forecast errors – measured by metrics such as MSE – associate with the transient dynamical properties of the system, characterized by two recently introduced indices: the instantaneous dimension (d), and the inverse persistence (θ). To quantify dynamical consistency, we analyze the dynamical discrepancy between forecasts and true states via standard error metrics applied to d and θ, e.g., MSEd and MSEθ. Similar to their traditional counterpart, MSEd and MSEθ yield non-negative values, with larger values indicating greater deviations from the true dynamics. In addition, we measure the signed error with respect to the true dynamics, aspect that is particularly useful in scenarios where the direction of the dynamical discrepancy (i.e., overestimation vs underestimation) have physical significance. Our analysis across three systems – the Lorenz system, the Kuramoto–Sivashinsky equation, and the Kolmogorov flow – reveals that states with larger d (higher dynamical complexity) and larger θ (lower persistence) tend to exhibit larger ML forecast errors, for both direct and recursive forecasting strategies. The results presented in this work provide a-priori estimates of where ML forecasts may under perform, information that is not readily accessible via standard error metrics. This information, in turn, can help guide ML training.