Validating emotion-driven multimodal AI for STEM vocational education: Evidence from student feedback

Abstract

This study examines the effectiveness and student perceptions of an emotion-driven multimodal artificial intelligence (AI) learning system designed for STEM-oriented vocational education. The system integrates real-time facial expression analysis (using a fine-tuned SSD MobileNetV1 model via face-api.js), speech prosody features, and behavioral interaction data to enable dynamic, emotion-aware instructional adaptation. Drawing on real-world system interaction data and survey responses from 300 vocational students enrolled in STEM-related programs (e.g., programming and technical training) in China, we employed SPSS for descriptive, reliability, factor, and correlation analyses. To validate the emotional intelligence of the system, a subset of the data was compared against human expert annotations, yielding an average emotion recognition F1-score of 0.86. Results demonstrated high internal consistency (Cronbach’s alpha = 0.94) and a three-factor model comprising emotional perception, adaptive instruction, and satisfaction/privacy trust. Open-ended responses revealed five thematic dimensions: personalized learning, emotional feedback, multimodal interaction, usability concerns, and improvement suggestions. These findings empirically validate the proposed AI model and offer actionable insights for designing emotion-aware adaptive learning systems in STEM vocational education.