DeepRetina: a multimodal framework for early diabetic retinopathy detection and progression prediction

Abstract

Diabetic retinopathy (DR) remains one of the top causes of vision loss globally, and early detection and accurate progression prediction are critical in its management. This paper introduces DeepRetina, a deep learning framework that integrates state-of-the-art multimodal retinal imaging techniques with patient-specific clinical data for the improved diagnosis and prognosis of DR. DeepRetina harnesses cutting-edge convolutional neural networks (CNNs) and attention mechanisms to jointly analyze optical coherence tomography (OCT) scans and fundus photographs. The architecture further includes a temporal module that investigates the longitudinal changes in the retina. DeepRetina fuses these heterogeneous data sources with patient clinical information in pursuit of early detection of DR and provides personalized predictions for the progression of the disease. We use a specially designed CNN architecture to process high-resolution retinal images, coupled with a self-attention mechanism that focuses on the most relevant features. This recurrent neural network (RNN) module empowers it to integrate time-series data that captures the evolution of retinal abnormalities. Another neural network branch considering patientspecific clinical data, such as demographic information, medical history, and laboratory test results, was taken into account and concatenated with the imaging features for a holistic analysis. DeepRetina achieved 95% sensitivity, 98% specificity for early DR detection, and a 0.92 area under the curve (AUC) for 5-year progression prediction, outperforming existing methods.