CLIP-based multimodal endorectal ultrasound enhances prediction of neoadjuvant chemoradiotherapy response in locally advanced rectal cancer

Abstract

Background Forecasting the patient’s response to neoadjuvant chemoradiotherapy (nCRT) is crucial for managing locally advanced rectal cancer (LARC). This study investigates whether a predictive model using image-text features extracted from endorectal ultrasound (ERUS) via Contrastive Language-Image Pretraining (CLIP) can predict tumor regression grade (TRG) before nCRT. Methods A retrospective analysis of 577 LARC patients who received nCRT followed by surgery was conducted from January 2018 to December 2023. ERUS scans and TRG were used to assess nCRT response, categorizing patients into good (TRG 0) and poor (TRG 1–3) responders. Image and text features were extracted using the ResNet50+RBT3 (RN50) and ViT-B/16+RoBERTa-wwm (VB16) components of the Chinese-CLIP model. LightGBM was used for model construction and comparison. A subset of 100 patients from each responder group was used to compare the CLIP method with manual radiomics methods (logistic regression, support vector machines, and random forest). SHapley Additive exPlanations (SHAP) technique was used to analyze feature contributions. Results The RN50 and VB16 models achieved AUROC scores of 0.928 (95% CI: 0.90–0.96) and 0.900 (95% CI: 0.86–0.93), respectively, outperforming manual radiomics methods. SHAP analysis indicated that image features dominated the RN50 model, while both image and text features were significant in the VB16 model. Conclusions The CLIP-based predictive model using ERUS image-text features and LightGBM showed potential for improving personalized treatment strategies. However, this study is limited by its retrospective design and single-center data.