Effect of large language models artificial intelligence chatgpt chatbot on achievement of computer education students

Abstract

In spite of the emergence of studies seeking to integrate chatbot into education, there is a wide literature gap in the Nigerian contexts. While most studies focus on the design and development of chatbots, there exists a very scarce literature on the effect of ChatGPT chatbot on students’ achievement. To address this gap, this study checked the effect of ChatGPT chatbot on computer education students. Using a repeated measures approach, 68 year one undergraduates participated in the study. A 50-item Computer Education Achievement Test (CEAT), which was subjected to expert validation and pilot study, was used to obtain data. The students were given two pre-test and posttest measures before and after the treatment. Data were computed using mixed design repeated measures analysis of variance and the findings revealed that the use of ChatGPT chatbot improved students’ performance (F (3,201) = 408.793, P = 0.000) with an effect size (ηp2) of 0.859. The finding also revealed no significant moderating influence of gender, age and experience on the effect of ChatGPT chatbot on students’ performance. The study has some implication for lecturers, researchers and policy experts on the need to incorporate ChatGPT chatbot in education given that it has the potential to improve students’ performance and equally promote self-paced individualized learning.