Evaluation of Medical Diagnosis Capabilities of Three Artificial Intelligence Models – ChatGPT-3.5, Google Gemini, Microsoft Copilot: Sustainable Development Goals (SDGs)

Abstract

Objectives: This study aims to assess and compare the diagnostic accuracy of three artificial intelligence (AI) models—ChatGPT-3.5, Microsoft Copilot, and Google Gemini—through their performance on clinical vignettes.   Theoretical Framework: Building on prior research into the application of AI in healthcare, particularly in diagnostic support, this study examines the potential of AI models to aid clinicians by providing accurate medical diagnoses, thus supporting decision-making in clinical contexts.   Methodology: A meta-analysis was conducted, followed by a comparative analysis using 34 clinical vignettes from Texas Tech University Health Sciences Center. Each AI model’s responses were evaluated for accuracy in diagnosing medical cases, and statistical significance was tested using the chi-square test.   Results and Discussion: ChatGPT-3.5 achieved the highest diagnostic accuracy (70.59%), outperforming Google Gemini (61.76%) and Microsoft Copilot (35.29%). ChatGPT-3.5 provided concise answers, while Google Gemini and Microsoft Copilot included disclaimers and additional recommendations. Chi-square analysis confirmed significant differences in performance, highlighting variations in diagnostic capabilities across models.   Research Implications: These findings underscore the importance of model selection when integrating AI into clinical workflows. AI models show promise in diagnostics but vary in approach and accuracy, warranting further refinement.   Originality/Value: This study is among the first to compare the diagnostic accuracy of ChatGPT-3.5, Google Gemini, and Microsoft Copilot, contributing valuable insights into AI’s application in healthcare diagnostics and supporting evidence for its potential role in enhancing patient care.