Multilabel Classification of Radiology Image Concepts Using Deep Learning

Abstract

Understanding and interpreting medical images, particularly radiology images, is a time-consuming task that requires specialized expertise. In this study, we developed a deep learning-based system capable of automatically assigning multiple standardized medical concepts to radiology images, leveraging deep learning models. These concepts are based on Unified Medical Language System (UMLS) Concept Unique Identifiers (CUIs) and describe the radiology images in detail. Each image is associated with multiple concepts, making it a multilabel classification problem. We implemented several deep learning models, including DenseNet121, ResNet101, and VGG19, and evaluated them on the ImageCLEF 2020 Medical Concept Detection dataset. This dataset consists of radiology images with multiple CUIs associated with each image and is organized into seven categories based on their modality information. In this study, transfer learning techniques were applied, with the models initially pre-trained on the ImageNet dataset and subsequently fine-tuned on the ImageCLEF dataset. We present the evaluation results based on the F1-score metric, demonstrating the effectiveness of our approach. Our best-performing model, DenseNet121, achieved an F1-score of 0.89 on the classification of the twenty most frequent medical concepts, indicating a significant improvement over baseline methods.