Leveraging search engines and llms for creative thinking: Effects of prompting skills, and domain knowledge

Abstract

Introduction. We investigated the effects of domain knowledge (DK) and prompting skills by leveraging a traditional search engine and a prompt-based LLM chatbot for a learning-oriented creative task. Method. On a simulated curated calligraphy exhibition task for which one must integrate DK and creativity to write summaries of curatorial planning, we observed behavior when using the NPM Collection, Google Search, and ChatGPT. We assessed the creative thinking demonstrated in the final summaries. User perception of tools supporting creative thinking was evaluated by the creativity support index. Analysis. We analysed information source usage behavior using descriptive statistical analysis and significance testing and used regression analysis to examine the predictive power of tool usage behavior, domain knowledge, and prompting skills for summary writing. Results. We discovered a positive predictive effect of DK and ChatGPT usage on the quality of curatorial planning summaries. Better prompting skills enhanced task performance and perceived creativity. For creative thinking, Google seemed to better foster exploration than ChatGPT. Conclusions. Prompt-based LLM chatbots support learning-oriented creative tasks. As prompting skills are essential, prompting should be integrated into information literacy education. The study provides implications for the design of search systems in human-information retrieval-LLM interaction to better support creative tasks.