Prompt again: How consistently useful are artificial intelligence chatbot responses when prompted with concerns about the reality of paediatric chronic pain?

Abstract

1 | INTRODUC TI ON The ubiquity and conversational ability of artificial intelligence (AI) chatbots have ushered in a new era. Individuals, including children, are increasingly turning to AI chatbots for interactive conversations, including those related to health. 1 Given this trend, current children might soon be classified as "AI-natives," growing up in a world where interacting with AI is a norm. This phenomenon raises questions about the consistency of health information provided by AI chatbots. 2 To date, the consistency of AI chatbot responses has been a subject of scrutiny. However, a notable gap exists in our understanding of this consistency in paediatric chronic pain, and the overall usefulness of AI chatbots for this population. This is a crucial area of inquiry because misconceptions and misunderstandings about chronic pain (such as questioning the reality of pain with no physical evidence of injury) are prevalent in society. 3 One common stigmatizing concern that children with chronic pain might express is the question, "Is it all in my head?," 4 and turn to AI for guidance. Therefore, this commentary aims to assess the usefulness and consistency of responses provided by common AI chatbots to this question. We accessed six popular AI chatbots on 2 January 2024; ChatGPT 4 Turbo, Google Bard (Gemini Pro) version 2023.12.18, Microsoft Copilot Creative Mode (GPT-4), Claude, Perplexity AI, and ChatGPT 3.5. Each chatbot was presented with the following prompt 10 times: "I am a child with chronic pain. Is it all in my head?". The usefulness of responses was analyzed based on a predefined scoring system of 10 criteria developed by the authors, deemed to contribute to a high-quality clinical response to our question. Appendix 1 provides definitions for each criterion and references to related literature for each. One point was given to each criterion met, so that scores ranged from 0 to 10 for each response. These response criteria included (1) sorry to hear, (2) actively addresses that pain is not in your head, (3) pain is multifactorial, (4) referral to health professional, (5) evidence-based, (6) not alone, (7) coaching tone, (8) asked clarifying questions, (9) child-friendly language (Flesch-Kincaid Grade Level <7), and (10) word count not too short or long (100-300 words). Chatbot responses were scored by two blinded raters independently (JWP and BS). Discrepancies in the responses were discussed, and if a decision could not be agreed upon, they were further adjudicated by SDT. A mean score, SD, and range were calculated for each chatbot based on the 10 responses. The readability of responses was evaluated by the range of the Flesch-Kincaid Grade Level score using Microsoft Word. The consistency of the two raters was calculated for each criterion using prevalence-adjusted and bias-adjusted kappa (PABAK). Figure 1 contains a summary of the usefulness and readability scores for each chatbot. PABAK values for each scoring criterion This is an open access article under the terms of the Creative Commons Attribution License, which permits use, distribution and reproduction in any medium, provided the original work is properly cited.