A friend or a foe? Evaluating ChatGPT's Impact on Students' Computational Thinking Skills

Abstract

With the easy accessibility of generative AI tools, software engineering students are increasingly using these tools in their coursework. However, the impact of these tools on the students' development of core computational thinking skills remains unclear. In this paper, we present our ongoing research that investigates whether access to ChatGPT during the learning process influences students' understanding and retention of essential computational skills. We specifically look at their ability to decompose complex problems (decomposition) and abstract key elements into classes and objects (abstraction). In a controlled, between-subject experiment, we divide students into two groups: one with ChatGPT assistance during the learning phase and a control group relying solely on traditional resources. Participants first undergo a study session to learn and apply decomposition and abstraction skills through various tasks, followed by a retention test session within a week from their study session. We measure the time taken to complete the tasks as well as the correctness of the solutions submitted by participants for each task. Participants also complete a post-study survey to provide insights into their experiences and to gauge their confidence in their solutions and acquired skills. We also analyze screen recordings and ChatGPT interaction logs to better understand the participants' usage strategies and interaction patterns. Our study aims to contribute to the ongoing discussion regarding the pedagogical role of generative AI in Software Engineering education and to offer guidance on effectively integrating these tools into teaching and learning practices.