Ridge Regression and the Elastic Net: How Do They Do as Finders of True Regressors and Their Coefficients?

Abstract

For the linear model (Formula presented.), where the number of regressors ( (Formula presented.) ) exceeds the number of observations ( (Formula presented.) ), the Elastic Net (EN) was proposed, in 2005, to estimate (Formula presented.). The EN uses both the Lasso, proposed in 1996, and ordinary Ridge Regression (RR), proposed in 1970, to estimate (Formula presented.). However, when (Formula presented.), using only RR to estimate (Formula presented.) has not been considered in the literature thus far. Because RR is based on the least-squares framework, only using RR to estimate (Formula presented.) is computationally much simpler than using the EN. We propose a generalized ridge regression (GRR) algorithm, a superior alternative to the EN, for estimating (Formula presented.) as follows: partition (Formula presented.) from left to right so that every partition, but the last one, has 3 observations per regressor; for each partition, we estimate (Formula presented.) with the regressors in that partition using ordinary RR; retain the regressors with statistically significant (Formula presented.) -ratios and the corresponding RR tuning parameter (Formula presented.), by partition; use the retained regressors and (Formula presented.) values to re-estimate (Formula presented.) by GRR across all partitions, which yields (Formula presented.). Algorithmic efficacy is compared using 4 metrics by simulation, because the algorithm is mathematically intractable. Three metrics, with their probabilities of RR’s superiority over EN in parentheses, are: the proportion of true regressors discovered (99%); the squared distance, from the true coefficients, of the significant coefficients (86%); and the squared distance, from the true coefficients, of estimated coefficients that are both significant and true (74%). The fourth metric is the probability that none of the regressors discovered are true, which for RR and EN is 4% and 25%, respectively. This indicates the additional advantage RR has over the EN in terms of discovering causal regressors.