Assessing the quality of ChatGPT responses to dementia caregivers’ questions

Abstract

Background: AI such as ChatGPT holds great promise to improve dementia patients’ and caregivers’ quality of life by providing high-quality responses to their questions about typical dementia behaviors. So far, however, evidence on the quality of such ChatGPT responses is limited. A few recent publications have investigated the quality of ChatGPT responses in other health conditions, however, our study is the first that used real-world questions asked by dementia caregivers themselves. Objectives: This pilot study examines the potential of ChatGPT to provide high-quality information that may enhance dementia care and patient-caregiver education. Methods: Our interprofessional team used a formal rating scale (scoring range: 0-5; the higher the score, the better the quality) to evaluate ChatGPT responses to real-world questions posed by dementia caregivers. We selected 60 posts by dementia caregivers from Reddit, a popular social media platform. These posts were verified by 3 interdisciplinary dementia clinicians as representing dementia caregivers’ desire for information in the areas of memory loss and confusion, aggression and driving. Word count for posts in the memory loss and confusion category ranged from 71 to 531 (mean: 218; median: 188), aggression posts ranged from 58-602 words (mean: 254; median:200) and driving posts ranged from 93 to 550 words (mean: 272; median: 276). Results: ChatGPT response quality scores ranged from 3 to 5; overall, 26 (43%) of the 60 responses received 5 points; 21 (35%), 4 points; and 13 (21.7%), 3 points, suggesting high quality. ChatGPT obtained consistently high scores synthesizing information to provide follow-up recommendations (96%), with the lowest scores in the area of comprehensiveness (63%). Conclusions: ChatGPT provided high-quality responses to complex questions posted by dementia caregivers, but it did have limitations. ChatGPT was unable to anticipate future problems that a human professional might recognize and address in a clinical encounter. At other times, ChatGPT recommended a strategy that the caregiver had already explicitly tried. This pilot study suggests the potential of AI to provide high-quality information to enhance dementia care and patient-caregiver education in tandem with information provided by licensed health care professionals. Evaluating the quality of responses is necessary to ensure that caregivers can make informed decisions. ChatGPT has the potential to transform healthcare practice by shaping how caregivers receive health information.