Prior knowledge-guided multimodal deep learning system for biomarker exploration and prognosis prediction of urothelial carcinoma

Abstract

Accurate survival prediction for urothelial carcinoma (UC) is critical but limited by a lack of AI systems that integrate multimodal data with prior knowledge. To address this gap, we developed a multimodal deep learning system that integrates histopathology, radiology, and structured pathology text. We incorporated prior knowledge to improve tumor segmentation and create knowledge-guided slide representations. CTContextNet and MacroContextNet were then employed for capturing radiological and macroscopic prognostic information. IM-NCTNet integrated multi-modality information for enhanced prediction. Across multi-center, large-scale, multi-cohort validation, the knowledge-guided prognostic system demonstrated higher performance over single-modality models, achieving C-index scores ranging from 0.809 to 0.867. The framework identified novel prognostic biomarkers related to infiltration patterns in muscle and renal parenchyma. High-risk Coloc_M, Coloc_R, and IMTS groups exhibited increased mortality risk, with hazard ratio values ranging from 2.47 to 16.38. The proposed AI framework offers a comprehensive and robust tool for UC prognosis, supporting refined patient management.