A Dual-Layer, Content-Aware Framework to Validate Online Student Engagement via ML-Based Comprehension Assessment

Abstract

Online student engagement monitoring tools based on computer vision and artificial intelligence are increasingly used in virtual classrooms to assess attentiveness through facial orientation, gaze, and posture. However, these systems largely capture superficial visual cues and fail to validate actual cognitive engagement or learning comprehension. This paper proposes a dual-layer, content-aware framework that verifies behavioural engagement scores (E-scores) using comprehension-based validation through lecture-specific quizzes. The framework integrates Whisper for real-time transcription and T5 for automatic generation of concise, content-aligned multiple-choice questions. Experiments were conducted across ten live lectures involving one hundred undergraduate students. Continuous engagement scores (0–100) received through a commercial system of engagement monitoring were in comparison with comprehension scores (C-scores) based on post-lecture quizzes using categorical thresholds: Low (0–49), Moderate (50–74), and High (75–100). The analysis discloses a low correlation between both E-, C-scores, and the prevalence of cases of mismatch (e.g., High-E/Low-C), where the visual attention was dissenting with the real. Some of the factors include cognitive overload, off-camera activities and partial occlusion. The modular pipeline runs more efficiently on a GPU-enabled workstation with CPU fallback support. Combining the understanding-based evaluation with the behavioural analytics, the proposed system enhances the validity of engagement measurement and enables the adaptive pedagogical practices in the online learning setting.