Sparsity in penalized empirical risk minimization

Abstract

Let (X,Y ) be a random couple in S × T with unknown distribution P. Let (X1,Y1), . . . , (Xn,Yn) be i.i.d. copies of (X,Y ), Pn being their empirical distribution. Let h1, . . . , hN : S → [-1, 1] be a dictionary consisting of N functions. For λ ∈ ℝN, denote f λ := ∑j=1N λj hj . Let ℓ : T × ℝ → ℝ be a given loss function, which is convex with respect to the second variable. Denote (ℓ • f )(x, y) := ℓ(y; f (x)). We study the following penalized empirical risk minimization problem λ̂ ε := argmin λ∈ℝN [Pn(ℓ • f λ)+ ε→λ→ℓpp] , which is an empirical version of the problem λ ε := argmin λ∈ℝN [Pn(ℓ • f λ)+ ε→λ→ℓpp] (here ε ≥ 0 is a regularization parameter; λ0 corresponds to ε = 0). A number of regression and classification problems fit this general framework. We are interested in the case when p ≥ 1, but it is close enough to 1 (so that p - 1 is of the order 1/logN , or smaller). We show that the "sparsity" of λε implies the "sparsity" of λ̂ε and study the impact of "sparsity" on bounding the excess risk P(ℓ • f̂λε ) - P(ℓ • fλ0 ) of solutions of empirical risk minimization problems. © 2009 Association des Publications de l'Institut Henri Poincaré.