Comparison of different methods for testing the significance of classification efficiency

Abstract

This paper discusses three methods to test the significance of classification efficiency (i.e. the fraction of correctly classified patterns of a test set), which can be applied to multiple class problems: the exact probability test, the Monte-Carlo test and the χ2 test. First, a short theoretical description of the three methods is given. The methods have been applied to different classification problems. A comparison is made in terms of the following criteria: required assumptions, power and time behaviour. To conclude, the paper describes a set of criteria for the selection of the appropriate classification efficiency testing method.