GPT-4-based AI agents—the new expert system for detection of antimicrobial resistance mechanisms?

Abstract

The European Committee on Antimicrobial Susceptibility Testing (EUCAST) recommends two steps for detecting beta-lactamases in Gram-negative bacteria. Screening for potential extended-spectrum beta-lactamase (ESBL), plasmid-mediated AmpC beta-lactamase, or carbapenemase production is confirmed. We aimed to validate generative pre-trained transformer (GPT)-4 and GPT-agent for pre-classification of disk diffusion to indicate potential beta-lactamases. We assigned 225 Gram-negative isolates based on phenotypic resistances against beta-lactam antibiotics and additional tests to one or more resistance mechanisms as follows: “none,” “ESBL,” “AmpC,” or “carbapenemase.” Next, we customized a GPT-agent with EUCAST guidelines and breakpoint table (v13.1). We compared routine diagnostics (reference) to those of (i) EUCAST-GPT-expert, (ii) microbiologists, and (iii) non-customized GPT-4. We determined sensitivities and specificities to flag suspect resistances. Three microbiologists showed concordance in 814/862 (94.4%) phenotypic categories and were used in median eight words (interquartile range [IQR] 4–11) for reasoning. Median sensitivity/specificity for ESBL, AmpC, and carbapenemase were 98%/99.1%, 96.8%/97.1%, and 95.5%/98.5%, respectively. Three prompts of EUCAST-GPT-expert showed concordance in 706/862 (81.9%) categories but were used in median 158 words (IQR 140–174) for reasoning. Sensitivity/specificity for ESBL, AmpC, and carbapenemase prediction were 95.4%/69.23%, 96.9%/86.3%, and 100%/98.8%, respectively. Non-customized GPT-4 could interpret 169/862 (19.6%) categories, and 137/169 (81.1%) agreed with routine diagnostics. Non-customized GPT-4 was used in median 85 words (IQR 72–105) for reasoning. Microbiologists showed higher concordance and shorter argumentations compared to GPT-agents. Humans showed higher specificities compared to GPT-agents. GPT-agent’s unspecific flagging of ESBL and AmpC potentially results in additional testing, diagnostic delays, and higher costs. GPT-4 is not approved by regulatory bodies, but validation of large language models is needed.