The Generalization Paradox of Ensembles

  • Elder IV J
  • 38


    Mendeley users who have this article in their library.
  • 15


    Citations of this article.


Ensemble models-built by methods such as bagging, boosting, and Bayesian model averaging-appear dauntingly complex, yet tend to strongly outperform their component models on new data. Doesn't this violate "Occam's razor"-the widespread belief that "the simpler of competing alternatives is preferred"? We argue no: if complexity is measured by function rather than form-for example, according to generalized degrees of freedom (GDF)-the razor's role is restored. On a two-dimensional decision tree problem, bagging several trees is shown to actually have less GDF complexity than a single component tree, removing the generalization paradox of ensembles.

Author-supplied keywords

  • Bagging
  • Complexity
  • Decision trees
  • Generalized degrees of freedom
  • Occam's razor

Get free article suggestions today

Mendeley saves you time finding and organizing research

Sign up here
Already have an account ?Sign in

Find this document


  • John F. Elder IV

Cite this document

Choose a citation style from the tabs below

Save time finding and organizing research with Mendeley

Sign up for free