Comparing lesson plan-driven and ask-me-anything chatbots: teaching a UNIX Shell course

Abstract

GPT-based models have enabled the creation of natural language chatbots that support both Inquiry-Based and Structured Learning approaches. This study offers a direct comparison of these two paradigms within a UNIX Shell scripting course by means of two chatbots: a Lesson Plan-Driven chatbot that ensures all students cover the same topics systematically, and an Ask-Me-Anything (AMA) chatbot more suited to exploratory learning. We compared two particular chatbots—Harvard’s CS50 (as an AMA chatbot) and OS315 (ours), as a lesson plan-driven chatbot—through four surveys. Results show a Net Promoter Score (NPS) of + 45 CI[25, 66] (N = 55) for the least performing lesson plan-driven chatbot and + 35 CI[11, 58.5] (N = 40) for the AMA chatbot. The majority of students favored a blend of human and chatbot instruction. Additionally, we discuss factors such as cost, accessibility, and why the same chatbot architecture, when applied to a Data Visualization course yields a lower NPS of + 9 CI[−6, + 23].