CXNet-m2: A Deep Model with Visual and Clinical Contexts for Image-Based Detection of Multiple Lesions

Abstract

Diagnosing multiple lesions on images is facing with challenges of incomplete and incorrect disease detection. In this paper, we propose a deep model called CXNet-m2 for the detection of multiple lesions on chest X-ray images. In our model, there is a convolutional neural network (CNN) for encoding the images, a recurrent neural network (RNN) for generating the next word (the name of lesion) and an attention mechanism to align the visual contexts with the prediction of words. There are two main contributions of CXNet-m2 to improve the work efficiency and increase the diagnosis accuracy. (1) Inspired by image captioning, CXNet-m2 adapts the classification system to a language model, where Bi-LSTM is used to learn the clinical relationship between lesions. (2) Inspired by attention mechanism, the prediction of possible lesions is guided by visual contexts, where the visual contexts are selected by the previously generated words and chosen visual regions. The experimental results on Chestx-ray14 show that CXNet-m2 achieves better AUC and the different versions of CXNet-m2 illustrate the importance of pre-training and clinical contexts.