Cross-Validation, Stacking and Bi-Level Stacking: Meta-Methods for Classification Learning

Abstract

So many methods have been invented for the problem of classification learning that practitioners now face a meta-level problem of choosing among alternatives or arbitrating between them. A natural idea is to use cross-validation to select one of several learning algorithms. A more general approach is Wolpert’s stacking, which uses the predictions of several methods together; and this may be further generalized to bi-level stacking, in which ordinary attributes are allowed to play a role in arbitrating between methods. In this paper, we examine cross-validation, stacking and bi-level stacking and present empirical results to illustrate key points.