FIRST: Combining forward iterative selection and shrinkage in high dimensional sparse linear regression

Abstract

We propose a new class of variable selection techniques for regression in high dimensional linear models based on a forward selection version of the LASSO, adaptive LASSO or elastic net, respectively to be called as forward iterative re-gression and shrinkage technique (FIRST), adaptive FIRST and elastic FIRST. These methods seem to work effectively for extremely sparse high dimensional linear models. We ex-ploit the fact that the LASSO, adaptive LASSO and elastic net have closed form solutions when the predictor is one-dimensional. The explicit formula is then repeatedly used in an iterative fashion to build the model until convergence occurs. By carefully considering the relationship between es-timators at successive stages, we develop fast algorithms to compute our estimators. The performance of our new es-timators are compared with commonly used estimators in terms of predictive accuracy and errors in variable selection.