Deep multimodal classification of image types in biomedical journal figures

Abstract

This paper presents a robust method for the classification of medical image types in figures of the biomedical literature using the fusion of visual and textual information. A deep convolutional network is trained to discriminate among 31 image classes including compound figures, diagnostic image types and generic illustrations, while another shallow convolutional network is used for the analysis of the captions paired with the images. Various fusion methods are analyzed as well as data augmentation approaches. The proposed system is validated on the ImageCLEF 2013 and 2016 figure and subfigure classification tasks, largely improving the currently best performance from 83.5% to 93.7% accuracy and 88.4% to 89.0% respectively.