A General Theory of Pathwise Coordinate Optimization

Abstract

Block coordinate descent algorithms have been extensively applied to convex and nonconvex statistical learning problems. However, there exist no adequate theory to justify theirt superior empirical performance. For convex problems, existing literature only shows sublinear rates of convergence to global optima, which is much slower than the observed empirical performance. For nonconvex problems, existing literature hardly provides any theoretical guarantees. To bridge this gap, we propose a unified computational framework, named PICASSO (Pathwise Calibrated Sparse Shooting algorithm). A major difference between PICASSO and previous work is that PICASSO exploits three new selection rules for active set identification. These rules ensure the algorithms to maintain sparse solutions throughout all iterations and allows us to establish linear rates of convergence to a unique sparse local optima with good statistical properties (e.g. minimax optimality and oracle properties) for PICASSO. We provide two concrete examples on sparse linear regression and logistic regression and establish new theoretical results on both parameter estimation and support recovery. Numerical experiments are presented to support our theory. An R package picasso implementing the proposed procedure is available on the Comprehensive R Archive Network http://cran.r-project.org/web/packages/picasso/.