Hyper-Heuristic Evolutionary Approach for Constructing Decision Tree Classifiers

Abstract

Decision tree models have earned a special status in predictive modeling since these are considered comprehensible for human analysis and insight. Classification and regression tree (CART) algorithm is one of the renowned decision tree induction algorithms to address classification as well as regression problems. Finding optimal values for the hyper parameters of a decision tree construction algorithm is a challenging issue. While making an effective decision tree classifier with high accuracy and comprehensibility, there is a need to address the question of setting optimal values for its hyper parameters like the maximum size of the tree, the minimum number of instances required in a node for inducing a split, node splitting criterion, and the amount of pruning. The hyper parameter setting influences the performance of the decision tree model. As known by researchers, there is no single setting of hyper parameters that works equally well for different datasets. A particular setting that gives an optimal decision tree for one dataset may produce a sub-optimal decision tree model for another dataset. In this paper, a hyper-heuristic approach was presented for tuning the hyper parameters of recursive and partition trees (Rpart), which is a typical implementation of CART in statistical and data analytics package R. The study employed an evolutionary algorithm as hyper-heuristic for tuning the hyper parameters of the decision tree classifier. The approach was named as hyper-heuristic evolutionary approach with recursive and partition trees (HEARpart). The proposed approach was validated on 30 datasets. It was statistically proven that HEARpart performed significantly better than WEKA’s J48 algorithm in terms of error rate, F-measure, and tree size. Furthermore, the suggested hyper-heuristic algorithm constructed significantly comprehensible models as compared to WEKA’s J48, CART, and other similar decision tree construction strategies. The results showed that the accuracy achieved by the hyper-heuristic approach was slightly less as compared to the other comparative approaches.