When doctors are trained to diagnose a specific disease, they learn faster when presented with cases in order of increasing difficulty. This creates the need for automatically estimating how difficult it is for doctors to classify a given case. In this paper, we introduce methods for estimating how hard it is for a doctor to diagnose a case represented by a medical image, both when ground truth difficulties are available for training, and when they are not. Our methods are based on embeddings obtained with deep metric learning. Additionally, we introduce a practical method for obtaining ground truth human difficulty for each image case in a dataset using self-assessed certainty. We apply our methods to two different medical datasets, achieving high Kendall rank correlation coefficients on both, showing that we outperform existing methods by a large margin on our problem and data.
CITATION STYLE
Hannemose, M. R., Sundgaard, J. V., Ternov, N. K., Paulsen, R. R., & Christensen, A. N. (2022). Was that so Hard? Estimating Human Classification Difficulty. In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics) (Vol. 13540 LNCS, pp. 88–97). Springer Science and Business Media Deutschland GmbH. https://doi.org/10.1007/978-3-031-17721-7_10
Mendeley helps you to discover research relevant for your work.