The weight of penalty optimization for ridge regression

Abstract

A method of weight optimization is introduced when fitting penalized ridge regression models. A penalty term added to a likelihood may be viewed in the light of a hierarchical likelihood. Under this context a method to estimate the variance of a random effect in a mixed model can be employed to obtain an estimate of the penalization weight.We review the theory of ridge penalties from a Bayesian point of view and show how an algorithm for estimating the variance of a random effect can be combined with hierarchical likelihood. The method is compared with other commonly used methods to obtain a penalty weight, such as leave-one-out cross validation, generalized cross validation, penalized quasi-likelihood methods and principal components estimation. Simulation studies are performed to compare the different approaches. For each of the methods we use packages already publicly available in the statistical software R.