On the talent vs. luck-based evaluation of the classification process

Abstract

Performance measures of classification algorithms play a crucial role in the evaluation of the learned models. Nevertheless, the vast majority of such measures are based on the same notion of classification performance, i.e., the ability of the classifier to recognize data samples from predefined training and testing sets. In this paper, we aim at introducing a new framework of evaluation of the classification process based not only on the aforementioned ability ('Talent' of the classifier) but also on randomness ('Luck') that would affect its performance. Based on the studies with socio-economic contexts where 'Luck' has been shown to play a crucial role in success and failure, we define a new measure to quantify the Talent versus Luck (TvL) tradeoff within a classification framework and prove its relationship with the generalization error. The proposed measure is validated via convolutional neural networks both with and without dropout layer, in order to highlight the relation of the measure to the generalization aspect, using the MNIST dataset. The experimental results confirm the fundamental role of TvL tradeoff in the evaluation of classifiers and in selecting the most 'successful' ones suggesting the TvL measure as a new, useful tool in the arsenal of evaluation of the classification process.