Variable Selection

We have already remarked that multicollinearity, i.e., nearly linearly Multicollinearity dependent columns in the design matrix, may increase the variance of the estimators $$\hat{\beta }_{i}$$ . For simplicity of presentation, we will assume throughout this section that the response is centered and predictor variables are standardized. More formally, Zvára (2008, Theorem 11.1 ) observes in the linear model ( 8.1 ) that $$\displaystyle{\mathop{\mathrm{\mathsf{E}}}\nolimits \|\hat{\beta }\|^{2} =\|\beta \| ^{2} +\sigma ^{2}\mathop{ \mathrm{\text{tr}}}\nolimits (\mathcal{X}^{\top }\mathcal{X})^{-1}}$$ and $$\displaystyle{\mathop{\mathrm{\mathsf{E}}}\nolimits \|\hat{Y }\|^{2} =\| \mathcal{X}\beta \|^{2} +\sigma ^{2}\mathop{ \mathrm{\text{rank}}}\nolimits (\mathcal{X}).}$$ It follows that multicollinearity does not affect the fitted values $$\hat{Y } = \mathcal{X}\hat{\beta }$$ Fitted values because the expectation of its squared length depends only on σ 2 and the rank of the model matrix $$\mathcal{X}$$ . On the other hand, the expectation of the squared length of the estimator $$\hat{\beta }$$ depends on the term $$\mathop{\mathrm{\text{tr}}}\nolimits (\mathcal{X}^{\top }\mathcal{X})^{-1} =\sum \lambda _{ i}^{-1}$$ , where λ i are the eigenvalues of $$\mathcal{X}^{\top }\mathcal{X}$$ . If the columns of $$\mathcal{X}$$ are nearly dependent, some of these eigenvalues may be very small and $$\mathop{\mathrm{\mathsf{E}}}\nolimits \|\hat{\beta }\|^{2}$$ then may become very large even if, technically, the design matrix still has full rank.