A comparative study of Bagging, Boosting and C4.5: The recent improvements in decision tree learning algorithm

Abstract

Decision tree learning algorithm has been successfully used in expert systems in capturing knowledge and presents a powerful method of inferring classification rules from a set of labeled examples. ID3 is a well known and the most basic decision tree-learning algorithm that is based on information gain theory. Improvements are made to this decision tree induction algorithm by Quinlan's C4.5 algorithm that uses gain ratio as opposed to information gain. Breiman's Bagging and Freund and Schapire's Boosting are recent methods of improving the predictive power of any classifier learning system. Both form a set of classifiers that are combined by voting, bagging by generating samples with replication of the data and boosting by adjusting the weights of training instances. In this research work both bagging and boosting have been applied to C4.5 algorithm and the corresponding predictive accuracies are computed by testing on a representative dataset. While both approaches substantially improve predictive accuracy of C4.5, boosting shows the greater benefit.