ChatGPT, Google, or PINK? Who Provides the Most Reliable Information on Side Effects of Systemic Therapy for Early Breast Cancer?

Abstract

Introduction: The survival in early breast cancer (BC) has been significantly improved thanks to numerous new drugs. Nevertheless, the information about the need for systemic therapy, especially chemotherapy, represents an additional stress factor for patients. A common coping strategy is searching for further information, traditionally via search engines or websites, but artificial intelligence (AI) is also increasingly being used. Who provides the most reliable information is now unclear. Material and Methods: AI in the form of ChatGPT 3.5 and 4.0, Google, and the website of PINK, a provider of a prescription-based mobile health app for patients with BC, were compared to determine the validity of the statements on the five most common side effects of nineteen approved drugs and one drug with pending approval (Ribociclib) for the systemic treatment of BC. For this purpose, the drugs were divided into three groups: chemotherapy, targeted therapy, and endocrine therapy. The reference for the comparison was the prescribing information of the respective drug. A congruence score was calculated for the information on side effects: correct information (2 points), generally appropriate information (1 point), and otherwise no point. The information sources were then compared using a Friedmann test and a Bonferroni-corrected post-hoc test. Results: In the overall comparison, ChatGPT 3.5 received the best score with a congruence of 67.5%, followed by ChatGPT 4.0 with 67.0%, PINK with 59.5%, and with Google 40.0% (p < 0.001). There were also significant differences when comparing the individual subcategories, with the best congruence achieved by PINK (73.3%, p = 0.059) in the chemotherapy category, ChatGPT 4.0 (77.5%; p < 0.001) in the targeted therapy category, and ChatGPT 3.5 (p = 0.002) in the endocrine therapy category. Conclusions: Artificial intelligence and professional online information websites provide the most reliable information on the possible side effects of the systemic treatment of early breast cancer, but congruence with prescribing information is limited. The medical consultation should still be considered the best source of information.