Trend and Seasonality Model Learning with Least Squares

In this chapter, a specific attention is paid to the determination of parametric models of the time series deterministic components: the trend and the seasonal patterns. In order to reach this goal, least squares data fitting methods are favored in this chapter. By least squares data fitting, we mean solutions that determine the unknown parameters of a parametric mathematical function by minimizing the sum of squared differences between the observed data and the fitted outputs provided by the model. Herein, it is more precisely shown that: The linear least squares approach should be considered when the model to determine is linear with respect to the unknown parameters. When N > n θ $$N > n_\theta $$ and rank ( Φ ) $$\text{rank}{(\boldsymbol {\varPhi })}$$ , the linear least squares solution is analytically equivalent to the solution to the normal equations Φ ⊤ Φ θ = Φ ⊤ y $$\boldsymbol {\varPhi }^\top \boldsymbol {\varPhi } \boldsymbol {\theta } = \boldsymbol {\varPhi }^\top \boldsymbol {y}$$ . The linear least squares solution is unique if N ≥ n θ $$N \geq n_\theta $$ and Φ $$\boldsymbol {\varPhi }$$ has full rank. If Φ $$\boldsymbol {\varPhi }$$ has full rank and has the QR factorization Φ = Q R $$\boldsymbol {\varPhi } = \boldsymbol {Q} \boldsymbol {R}$$ , then the linear least squares solution can be found by back substitution R θ = Q ⊤ y $$\boldsymbol {R} \boldsymbol {\theta } = \boldsymbol {Q}^\top \boldsymbol {y}$$ . By using the SVD of Φ $$\boldsymbol {\varPhi }$$ , the minimum norm solution of the linear least squares problem is given by V Σ † U ⊤ y = ∑ i = 1 r u i ⊤ y σ i v i $$\boldsymbol {V} \boldsymbol {\varSigma }^\dagger \boldsymbol {U}^\top \boldsymbol {y} = \sum _{i = 1}^r \frac {\boldsymbol {u}_i^\top \boldsymbol {y}}{\sigma _i} \boldsymbol {v}_i$$ , where r = rank ( Φ ) $$r = \text{rank}{(\boldsymbol {\varPhi })}$$ . The SVD of Φ $$\boldsymbol {\varPhi }$$ is the tool to use to determine the condition number of Φ $$\boldsymbol {\varPhi }$$ . When the initial linear least squares problem is ill conditioned, the user can either change the basis functions composing Φ $$\boldsymbol {\varPhi }$$ or use a regularization approach to lead to better fits and less ill conditioned linear least squares problems. When nonlinear models are involved, iterative minimization algorithms must be used to determine the unknown parameter vector θ $$\boldsymbol {\theta }$$ . For nonlinear least squares problem, the Levenberg–Marquardt method (also known as the damped nonlinear least squares method) is the generic algorithm used in many software applications for solving curve fitting problems. Because the Levenberg–Marquardt method finds only a local minimum, a specific attention must be paid to the initialization in order to guarantee that, in the end, the estimated local minimum is the global minimum of the nonlinear least squares cost function. Resampling techniques such as the K-fold procedure can be used for ( i ) $$(i)$$ assessing how the results of a linear least squares minimization problem can generalize to an independent data set and ( i i ) $$(ii)$$ tuning the hyperparameters used in the linear least squares solutions. Model order selection can be performed by resorting to information criteria such as the famous AIC.