Pareto neuro-ensembles

Abstract

The formation of a neural network ensemble has attracted much attention in the machine learning literature. A set of trained neural networks are combined using a post-gate to form a single super-network. One main challenge in the literature is to decide on which network to include in, or exclude from the ensemble. Another challenge is how to define an optimum size for the ensemble. Some researchers also claim that for an ensemble to be effective, the networks need to be different. However, there is not a consistent definition of what “different” means. Some take it to mean weakly correlated networks, networks with different bias-variance trade-off, and/or networks which are specialized on different parts of the input space. In this paper, we present a theoretically sound approach for the formation of neural network ensembles. The approach is based on the dominance concept that determines which network to include/exclude, identifies a suitable size for the ensemble, and provides a mechanism for quantifying differences between networks. The approach was tested on a number of standard dataset and showed competitive results.