Automatic Interpretation of Brain Medical Images Using Hierarchical Classification and Image Captioning Model

Abstract

Brain imaging plays a crucial role in diagnosing neurological disorders. However, generating precise and detailed descriptions of brain images remains a significant challenge due to the anatomical complexity and the technical nature of medical terminology. This research proposes a novel approach that integrates hierarchical classification with image captioning to enhance the analysis of medical brain images. The method begins with a classification model designed to extract semantic features, such as modality type, anatomical orientation, and abnormality status. These features are then utilized to guide the captioning model, which employs a Vision Transformer (ViT) as an encoder and a pre-trained medical language model for decoder. By combining these techniques, the proposed approach aims to bridge the gap between visual data and textual interpretation. Experimental results demonstrate a 12.39% improvement in BLEU score performance, with average BLEU scores increasing from 0.2979 (without integration) to 0.3348 (with integration). These findings highlight the potential of combining hierarchical classification with captioning to address the limitations of traditional methods.