COVID-19 Pneumonia Classification with Transformer from Incomplete Modalities

Abstract

COVID-19 is a viral disease that causes severe acute respiratory inflammation. Although with less death rate, its increasing infectivity rate, together with its acute symptoms and high number of infections, is still attracting growing interests in the image analysis of COVID-19 pneumonia. Current accurate diagnosis by radiologists requires two modalities of X-Ray and Computed Tomography (CT) images from one patient. However, one modality might miss in clinical practice. In this study, we propose a novel multi-modality model to integrate X-Ray and CT data to further increase the versatility and robustness of the AI-assisted COVID-19 pneumonia diagnosis that can tackle incomplete modalities. We develop a Convolutional Neural Networks (CNN) and Transformers hybrid architecture, which extracts extensive features from the distinct data modalities. This classifier is designed to be able to predict COVID-19 images with X-Ray image, or CT image, or both, while at the same time preserving the robustness when missing modalities are found. Conjointly, a new method is proposed to fuse three-dimensional and two-dimensional images, which further increase the feature extraction and feature correlation of the input data. Thus, verified with a real-world public dataset of BIMCV-COVID19, the model outperform state-of-the-arts with the AUC score of 79.93%. Clinically, the model has important medical significance for COVID-19 examination when some image modalities are missing, offering relevant flexibility to medical teams. Besides, the structure may be extended to other chest abnormalities to be detected by X-ray or CT examinations. Code is available at https://github.com/edurbi/MICCAI2023.