Eyes on the Code: Mapping Critical Thinking Through Eye-Tracking for Student-LLM Coding Interactions

Abstract

The integration of Large Language Models (LLMs) into coding processes raises essential questions about their effect on critical thinking and programming abilities. Our study assessed 26 university students who provided 104 data samples of LLM use in their programming tasks. The students' coding processes were analyzed through eye-tracking, questionnaires, and interviews. The students' interaction with the LLM (ChatGPT) was also recorded and analyzed using Natural Language Processing techniques. Our quantitative and qualitative analyses reveal that when accounting for both task-specific (programming difficulty) and user-specific (student coding proficiency) variables, a student's self-confidence, reliance, and confidence in the LLM's answers significantly predict whether critical thinking occurs among students. Notably, higher confidence in the LLM's answers correlates with reduced critical thinking, while higher self-confidence in one's coding abilities correlates with increased critical thinking and reduced LLM usage. Our findings also indicate that LLM use transforms critical thinking, shifting it toward information verification, response integration, and coding task stewardship. These insights highlight emerging design challenges and opportunities for developing LLM tools better suited for programming in an academic coding environment.