The Multimodal Deep Learning for Diagnosing COVID-19 Pneumonia from Chest CT-Scan and X-Ray Images

Abstract

Due to the COVID-19 Pandemic, doctors need to make medical decisions for their patients based on many examinations (e.g., polymerase chain reaction test, temperature test, CT-Scans, or X-rays). However, transfer learning has been used in several researches and focuses on only a single modality of biomarkers (e.g., CT-Scan or X-Ray) for diagnosing Pneumonia. In recent studies, a single modality has its own classification accuracy and every different biomarker may provide complementary information for detecting COVID-19 Pneumonia. The COVID-19 virus can be detected by CT-Scan and X-Ray imaging of the chest. In this work, we propose to use concatenation of two different transfer learning models using an open-source dataset of 2500 CT-Scan images and 2500 X-ray images for classifying CT-Scan images and X-ray images into two classes: normal and COVID-19 Pneumonia. We have used DenseNet121, MobileNet, Xception, InceptionV3, ResNet50, and VGG16 models for image recognition in our work. As a result, we achieve the best classification accuracy of 99.87% of the concatenation of ResNet50 and VGG16 networks. We also achieved the best classification accuracy of 98.00% when using a single modality of CT-Scan ResNet50 networks and classification accuracy of 98.93% for X-Ray VGG16 networks. Our multimodal fusion method shows a better classification accuracy compared to the method of using a single modality of biomarkers.