Multi-transSP: Multimodal Transformer for Survival Prediction of Nasopharyngeal Carcinoma Patients

Abstract

Nasopharyngeal carcinoma (NPC) is a malignant tumor that often occurs in Southeast Asia and southern China. Since there is a need for a more precise personalized therapy plan that depends on accurate prognosis prediction, it may be helpful to predict patients’ overall survival (OS) based on clinical data. However, most of the current deep learning (DL) based methods which use a single modality fail to effectively utilize amount of multimodal data of patients, causing inaccurate survival prediction. In view of this, we propose a Multimodal Transformer for Survival Prediction (Multi-TransSP) of NPC patients that uses tabular data and computed tomography (CT) images jointly. Taking advantage of both convolutional neural network and Transformer, the architecture of our network is comprised of a multimodal CNN-Based Encoder and a Transformer-Based Encoder. Particularly, the CNN-Based Encoder can learn rich information from specific modalities and the Transformer-Based Encoder is able to fuse multimodal feature. Our model automatically gives the final prediction of OS with a concordance index (CI) of 0.6941 on our in-house dataset, and our model significantly outperforms other methods using any single source of data or previous multimodal frameworks. Code is available at https://github.com/gluglurice/Multi-TransSP.