Sparse recovery in convex hulls via entropy penalization1

Abstract

Let (X,Y ) be a random couple in S × T with unknown distribution P and (X1,Y1), . . . , (Xn,Yn) be i.i.d. copies of (X,Y ). Denote Pn the empirical distribution of (X 1,Y1), . . . , (Xn,Yn). Let h 1, . . . , hN : S & -1, 1 be a dictionary that consists of N functions. For λ ε RN, denote f λ := ∑Nj =1 λj hj . Let ℓ : T × ℝ & ℝ be a given loss function and suppose it is convex with respect to the second variable. Let (ℓ · f )(x, y) := ℓ(y;f (x)). Finally, let λ ⊂ℝN be the simplex of all probability distributions on {1, . . . , N}. Consider the following penalized empirical risk minimization problem [Mathamatical Exp.] along with its distribution dependent version [Mathamatical Exp.] where ε ≥ 0 is a regularization parameter. It is proved that the "approximate sparsity" of λε implies the "approximate sparsity" of λε and the impact of "sparsity" on bounding the excess risk of the empirical solution is explored. Similar results are also discussed in the case of entropy penalized density estimation.