Rate-oriented point-wise confidence bounds for ROC curves

Abstract

Common approaches to generating confidence bounds around ROC curves have several shortcomings. We resolve these weaknesses with a new 'rate-oriented' approach. We generate confidence bounds composed of a series of confidence intervals for a consensus curve, each at a particular predicted positive rate (PPR), with the aim that each confidence interval contains new samples of this consensus curve with probability 95%. We propose two approaches; a parametric and a bootstrapping approach, which we base on a derivation from first principles. Our method is particularly appropriate with models used for a common type of task that we call rate-constrained, where a certain proportion of examples needs to be classified as positive by the model, such that the operating point will be set at a particular PPR value. © 2014 Springer-Verlag.