Assessing Chat Generative Pretrained Transformer-4’s clinical utility in acute compartment syndrome: a comprehensive evaluation of accuracy, completeness, and readability

Abstract

OBJECTIVE: Artificial intelligence tools like Chat Generative Pretrained Transformer-4 are increasingly used in clinical decision-making, but their reliability for acute compartment syndrome remains understudied. The aim of the study was to evaluate Chat Generative Pretrained Transformer4’s accuracy, completeness, and quality in responding to acute compartment syndrome-related queries, addressing gaps in artificial intelligenceassisted medical information. METHODS: Chat Generative Pretrained Transformer-4 was given 60 questions (40 open-ended and 20 binary) that were taken from the American Academy of Orthopaedic Surgeons 2019 acute compartment syndrome guidelines. Responses were evaluated independently by two orthopedic specialists using the Quality Criteria for Consumer Health Information instrument (information quality), Flesch-Kincaid Reading Ease Score (readability), and Likert scales (accuracy and completeness). Inter-rater reliability (Cohen’s kappa) was a statistical analysis. RESULTS: Chat Generative Pretrained Transformer-4 demonstrated high accuracy (95% for both question types) and completeness (mean scores: 5.92±0.8 [accuracy], 2.9±0.5 [completeness]). DISCERN scores were “excellent” (69–72), though source reliability was limited. Readability was “very difficult” (Flesch-Kincaid Reading Ease Score: 22.19), potentially hindering patient comprehension. CONCLUSION: Although Chat Generative Pretrained Transformer-4 is excellent at providing precise, high-quality acute compartment syndrome information, its complicated language and lack of credible sources make it difficult for wider adoption. To improve clinical utility and patient education, readability and transparency must be given top priority in future artificial intelligence developments.