A simplified volume under the ROC hypersurface

Abstract

The Receiver Operator Characteristic plot allows a classifier to be evaluated and optimised over all possible operating points. The Area Under the ROC has become a standard performance evaluation criterion in two-class pattern recognition problems, used to compare different classification algorithms independently of operating points, priors, and costs. Extending this measure to the multiclass case is considered in this paper, called the volume under the ROC hypersurface. A simplified measure is derived that ignores specific intra-class dimensions, and regards inter-class performances only. It is shown that this measure generalises from the 2-class case, but the bounds between random and perfect classification differ, with the lower bound tending towards zero as the dirnensionality increases. A number of experiments with known distributions are used to verify the bounds, and to investigate a numerical integration approach to estimating the volume. Experiments on real data compare several competing in terms of both error-rate and the volume. It was found that sonic classifiers compete in terms of error-rate, but have significantly different volume scores, illustrating the importance of the approach.