Diagnostic Performance of Multimodal Large Language Models in the Analysis of Oral Pathology

Abstract

OBJECTIVE: This study evaluated the accuracy and repeatability of ChatGPT-4o, a multimodal AI model, in interpreting photographs of oral mucosal lesions, and explored its potential as a diagnostic support tool for specialists and non-specialists. METHODS: Thirty clinical photographs of oral and labial mucosal lesions were analysed using ChatGPT-4o. For each image, 30 responses were generated across 20 days. The model was asked to identify the anatomical location, suggest a diagnosis, and recommend diagnostic tests and treatments. Two oral pathology experts assessed 3600 responses using a three-point scale (0 = incorrect, 1 = partially correct, 2 = correct). Accuracy and repeatability were analysed using accuracy rates, Gwet's AC and percent agreement. RESULTS: ChatGPT-4o achieved 71.4% accuracy in identifying lesion location and 58.2% in diagnosis. In cases with correct diagnoses, the model reached 90.7% and 95.8% accuracy in suggesting diagnostic tests and treatments, respectively. Repeated responses showed substantial to almost perfect agreement across all evaluated aspects. CONCLUSIONS: ChatGPT-4o showed potential as a reliable and accessible tool to support the initial assessment of oral lesions. Although not a substitute for clinical judgment, it may enhance diagnostic efficiency, particularly in resource-limited settings. Further validation is needed before clinical use.