Evaluation of large language models in female malignancy Q&A

Abstract

The future of artificial intelligence specialist doctors depends on foundational large language models (LLMs). However, there is currently a lack of evaluation studies on their Q&A capabilities in female malignancy. This cross-sectional study created a benchmark dataset consisting of 205 female malignancy Q&A to evaluate and compare the performance of 7 popular LLMs, with 3 human oncologists serving as a comparison. Only OpenAI o1, DeepSeek-R1, and Llama-3.1-405B achieved both accuracy and consistency rates exceeding 80%. Among these, Llama-3.1-405B demonstrated significantly higher accuracy than OpenAI o1 (P = .002) and potentially higher than DeepSeek-R1 (P = .059), with significantly lower latency (P < .001). After removing token limits, both OpenAI o1 and DeepSeek-R1 exhibited strong reasoning abilities, achieving accuracy rates of 98.5% and 95.6%, respectively, with no significant difference from Llama-3.1-405B (P > .05). The agreement between the models was fair, with an agreement rate of 0.318. Error analysis revealed that the highest error rates were associated with questions involving flexible medical scenarios and complex medical terminology. In female malignancy Q&A tasks, Llama-3.1-405B serves as a strong baseline model, while OpenAI o1 and DeepSeek-R1 show promise in inference, especially when token limitations are removed. However, their reasoning processes lack full consistency, likely due to differences in architectures and training methods. LLMs continue to face challenges in real-world medical scenarios and with complex medical terminology, highlighting the need for ongoing updates and the integration of clinical data to refine these models and improve their ability to address specialized medical needs.