Evaluation of Artificial Intelligence-Based Grading of Diabetic Retinopathy in Primary Care

Abstract

Importance: There has been wide interest in using artificial intelligence (AI)-based grading of retinal images to identify diabetic retinopathy, but such a system has never been deployed and evaluated in clinical practice. Objective: To describe the performance of an AI system for diabetic retinopathy deployed in a primary care practice. Design, Setting, and Participants: Diagnostic study of patients with diabetes seen at a primary care practice with 4 physicians in Western Australia between December 1, 2016, and May 31, 2017. A total of 193 patients consented for the study and had retinal photographs taken of their eyes. Three hundred eighty-six images were evaluated by both the AI-based system and an ophthalmologist. Main Outcomes and Measures: Sensitivity and specificity of the AI system compared with the gold standard of ophthalmologist evaluation. Results: Of the 193 patients (93 [48%] female; mean [SD] age, 55 [17] years [range, 18-87 years]), the AI system judged 17 as having diabetic retinopathy of sufficient severity to require referral. The system correctly identified 2 patients with true disease and misclassified 15 as having disease (false-positives). The resulting specificity was 92% (95% CI, 87%-96%), and the positive predictive value was 12% (95% CI, 8%-18%). Many false-positives were driven by inadequate image quality (eg, dirty lens) and sheen reflections. Conclusions and Relevance: The results demonstrate both the potential and the challenges of using AI systems to identify diabetic retinopathy in clinical practice. Key challenges include the low incidence rate of disease and the related high false-positive rate as well as poor image quality. Further evaluations of AI systems in primary care are needed.