A Comparative Study of ChatGPT and BingAI in Answering the National Eligibility Entrance Test for Postgraduates (NEET-PG)-Style Practice Questions: A Cross-Sectional Analysis

Abstract

PURPOSE: The integration of artificial intelligence (AI) into medical education has witnessed significant progress, particularly in the domain of language models. This study focuses on assessing the performance of two notable language models, ChatGPT and BingAI Precise, in answering the National Eligibility Entrance Test for Postgraduates (NEET-PG)-style practice questions, simulating medical exam formats. METHODS: A cross-sectional study conducted in June 2023 involved assessing ChatGPT and BingAI Precise using three sets of NEET-PG practice exams, comprising 200 questions each. The questions were categorized by difficulty levels (easy, moderate, difficult), excluding those with images or tables. The AI models' responses were compared to reference answers provided by the Dr. Bhatia Medical Coaching Institute (DBMCI). Statistical analysis was employed to evaluate accuracy, coherence, and overall performance across different difficulty levels. RESULTS: In the analysis of 600 questions across three test sets, both ChatGPT and BingAI demonstrated competence in answering NEET-PG style questions, achieving passing scores. However, BingAI consistently outperformed ChatGPT, exhibiting higher accuracy rates across all three question banks. The statistical comparison indicated significant differences in correct answer rates between the two models. CONCLUSIONS: The study concludes that both ChatGPT and BingAI have the potential to serve as effective study aids for medical licensing exams. While both models showed competence in answering questions, BingAI consistently outperformed ChatGPT, suggesting its higher accuracy rates. Future improvements, including enhanced image interpretation, could further establish these large language models (LLMs) as valuable tools in both educational and clinical settings.