ChatGPT Convincingly Explains Organic Chemistry Reaction Mechanisms Slightly Inaccurately with High Levels of Explanation Sophistication

Abstract

The chemistry education research community values and emphasizes the role of constructing explanations and mechanistic reasoning to support students’ learning of organic chemistry. Emerging large language model (LLM) and generative artificial intelligence (GAI) technologies are uniquely equipped to advance the teaching and learning of chemistry. GAI-based chatbots, such as ChatGPT, have the potential to help students learn mechanistic reasoning through their generated responses. This study investigates the extent to which 255 ChatGPT-generated responses are accurate explanations of 85 different reaction mechanisms and exhibit mechanistic reasoning as categorized by the levels of explanation sophistication framework. The study also explores the effects of prompt engineering on mechanism accuracy and explanation sophistication through three types of prompt cueing. Study findings show that (1) a quarter of responses are fully accurate explanations of reaction mechanisms and the majority contain predominantly accurate explanations of chemical phenomena and identification of nucleophiles and electrophiles, (2) responses exhibit high levels of explanation sophistication, and (3) prompt engineering plays a significant role in eliciting high levels of explanation sophistication but not mechanism description accuracy. Results are situated in mechanistic reasoning and prompt engineering frameworks with a focus on how these new technologies can be integrated into the chemistry classroom.