Human-in-the-Loop Performance of LLM-Assisted Arterial Blood Gas Interpretation: A Single-Center Retrospective Study

Abstract

Background and Objectives: Interpreting acid–base disorders is challenging, particularly in complex or mixed cases. Given the growing potential of large language models (LLMs) to assist in cognitively demanding tasks, this study evaluated their performance in interpreting arterial blood gas (ABG) results. Materials and Methods: In this single-center retrospective study, 200 ABG datasets were curated to include 40 cases in each of five diagnostic categories: metabolic acidosis, respiratory acidosis, metabolic alkalosis, respiratory alkalosis, and no acid–base disorder. Three medical students, each assigned to one LLM (ChatGPT GPT-4o, Copilot GPT-4, or Gemini 1.5-flash/2.5-flash), perform ABG interpretation using two evaluation methods: interpretation (LLM-I) and interpretation with supervision model (LLM-S). Two clinical pathologists independently performed the conventional evaluation to serve as the reference standard. Results: Agreement for identifying the primary acid–base (APD) disorder was strong across all approaches (Cohen’s κ ≥ 0.88). For identifying both primary and secondary disorders regardless of order (APSD), LLM-I showed moderate agreement (ChatGPT κ = 0.65, Copilot κ = 0.61, Gemini κ = 0.62), whereas LLM-S achieved strong agreement (ChatGPT κ = 0.91, Copilot κ = 0.81, Gemini κ = 0.81). Conclusions: LLM-assisted ABG interpretation demonstrates strong concordance with expert interpretation in detecting primary acid–base disorders. These tools may enhance the understanding of acid–base disorders while reducing calculation-related errors among medical students.