Convexity, classification, and risk bounds

  • Bartlett P
  • Jordan M
  • McAuliffe J
  • 284

    Readers

    Mendeley users who have this article in their library.
  • 400

    Citations

    Citations of this article.

Abstract

Many of the classification algorithms developed in the machine learning literature, including the support vector machine and boosting, can be viewed as minimum contrast methods that minimize a convex surrogate of the 0–1 loss function. The convexity makes these algorithms computationally efficient. The use of a surrogate, however, has statistical consequences that must be balanced against the computational virtues of convexity. To study these issues, we provide a general quantitative relationship between the risk as assessed using the 0–1 loss and the risk as assessed using any nonnegative surrogate loss function.We show that this relationship gives nontrivial upper bounds on excess risk under the weakest possible condition on the loss function—that it satisfies a pointwise form of Fisher consistency for classification. The relationship is based on a simple variational transformation of the loss function that is easy to compute in many applications.We also present a refined version of this result in the case of low noise, and show that in this case, strictly convex loss functions lead to faster rates of convergence of the risk than would be implied by standard uniform convergence arguments. Finally, we present applications of our results to the estimation of convergence rates in function classes that are scaled convex hulls of a finite-dimensional base class, with a variety of commonly used loss functions.

Author-supplied keywords

  • Boosting
  • Convex optimization
  • Empirical process theory
  • Machine learning
  • Rademacher complexity
  • Support vector machine

Get free article suggestions today

Mendeley saves you time finding and organizing research

Sign up here
Already have an account ?Sign in

Find this document

Authors

  • Peter L. Bartlett

  • Michael I. Jordan

  • Jon D. McAuliffe

Cite this document

Choose a citation style from the tabs below

Save time finding and organizing research with Mendeley

Sign up for free