Ridge Regression and the Elastic Net: How Do They Do as Finders of True Regressors and Their Coefficients?

For the linear model Y = X b + e r r o r , where the number of regressors ( p ) exceeds the number of observations ( n ), the Elastic Net (EN) was proposed, in 2005, to estimate b . The EN uses both the Lasso, proposed in 1996, and ordinary Ridge Regression (RR), proposed in 1970, to estimate b . However, when p > n , using only RR to estimate b has not been considered in the literature thus far. Because RR is based on the least-squares framework, only using RR to estimate b is computationally much simpler than using the EN. We propose a generalized ridge regression (GRR) algorithm, a superior alternative to the EN, for estimating b as follows: partition X from left to right so that every partition, but the last one, has 3 observations per regressor; for each partition, we estimate Y with the regressors in that partition using ordinary RR; retain the regressors with statistically significant t -ratios and the corresponding RR tuning parameter k , by partition; use the retained regressors and k values to re-estimate Y by GRR across all partitions, which yields b . Algorithmic efficacy is compared using 4 metrics by simulation, because the algorithm is mathematically intractable. Three metrics, with their probabilities of RR’s superiority over EN in parentheses, are: the proportion of true regressors discovered (99%); the squared distance, from the true coefficients, of the significant coefficients (86%); and the squared distance, from the true coefficients, of estimated coefficients that are both significant and true (74%). The fourth metric is the probability that none of the regressors discovered are true, which for RR and EN is 4% and 25%, respectively. This indicates the additional advantage RR has over the EN in terms of discovering causal regressors.