Many are better than one: Improving probabilistic estimates from decision trees

Abstract

Decision trees, a popular choice for classification, have their limitation in providing probability estimates, requiring smoothing at the leaves. Typically, smoothing methods such as Laplace or m-estimate are applied at the decision tree leaves to overcome the systematic bina introduced by the frequency-based estimates, In this work, wo show that an ensemble of decision trees significantly improves the quality of the probability estimates produced at the decision tree leaves. The ensembles overcomes the myopia of the leaf frequency based estimates. We show the effectiveness of the probabilistic decision trees as a part of the Predictive Uncertainty Challenge. We ulso include three additional highly imbalanced datasets in our study. We show that the ensemble methods significantly improve not, only the quality of the probability estimates but also the AUC for the irnbalanced datasets. © Springer-Verlag Berlin Heidelberg 2006.