Random Aggregated and Bagged Ensembles of SVMs: An Empirical Bias-Variance Analysis

Abstract

Bagging can be interpreted as an approximation of random aggregating, an ideal ensemble method by which base learners are trained using data sets randomly drawn according to an unknown probability distribution. An approximate realization of random aggregating can be obtained through subsampled bagging, when large training sets are available. In this paper we perform an experimental bias-variance analysis of bagged and random aggregated ensembles of Support Vector Machines, in order to quantitatively evaluate their theoretical variance reduction properties. Experimental results with small samples show that random aggregating, implemented through subsampled bagging, reduces the variance component of the error by about 90%, while bagging, as expected, achieves a lower reduction. Bias-variance analysis explains also why ensemble methods based on subsampling techniques can be successfully applied to large data mining problems. © Springer-Verlag 2004.