Investigating Users' Search Behavior and Outcome with ChatGPT in Learning-oriented Search Tasks

Abstract

Searching has become an essential method for acquiring knowledge. The field of Search as Learning (SAL) has traditionally explored how users engage with search engines for learning tasks, yet these engines frequently falter with complex cognitive challenges. The emergence of large language models (LLMs) like ChatGPT has addressed these shortcomings, marking a shift towards conversational search methods. Despite its potential, few research has investigated how ChatGPT supports users in the SAL context. To bridge this gap, we conducted a dedicated user study involving thirty-one undergraduates performing nine distinct learning-related search tasks. These tasks were divided into three topics, each explored at three levels of cognitive complexity. Using a Latin square design, we analyzed users' search behavior and outcome when using three search modes - traditional search engines, ChatGPT, and their combination - across various topics and cognitive complexities. The findings suggest that ChatGPT significantly boosts efficiency and enhances user experience and outcomes, particularly in more complex tasks. This research highlights the increasing importance of generative AI in enriching users' information-seeking endeavors.