The ROC Diagonal is Not Layperson’s Chance: A New Baseline Shows the Useful Area

Abstract

In many areas of our daily lives (e.g., healthcare), the performance of a binary diagnostic test or classification model is often represented as a curve in a Receiver Operating Characteristic (ROC) plot and a quantity known as the area under the ROC curve (AUC or AUROC). In ROC plots, the main diagonal is often referred to as “chance” or the “random line”. In general, however, this does not correspond to the layperson’s concept of chance or randomness for binary outcomes. Rather, this represents a special case of layperson’s chance, or the ROC curve for a classifier that has the same distribution of scores for the positive class and negative class. Where the ROC curve of a model deviates from the main diagonal, there is information. However, not all information is “useful information” compared to chance, including some areas and points above the diagonal. We define the binary chance baseline to identify areas and points in a ROC plot that are more useful than chance. In this paper, we explain this novel contribution about the state-of-art and provide examples that classify benchmark data.