Multimodal Emotion Recognition System for Mathematics Classrooms: Design, Performance, and Future Real-Time Applications

Abstract

Pattern recognition and artificial intelligence have made significant progress in the field of affective computing in recent years, especially in the application of multimodal data fusion, which shows strong potential. In educational scenarios, emotion recognition systems are able to optimize teaching strategies and enhance learning by analyzing students' emotional states. However, affective computing in the math classroom still faces significant challenges, such as the complexity and diversity of students' emotions, the high demands of real-time data processing, and the technical difficulties of multimodal data fusion. To cope with these issues, this study proposes a multimodal emotion recognition system that fuses three modal data, namely, facial expression, tone of voice, and text input, with simultaneous data acquisition. The system is based on a hybrid architecture of Convolutional Neural Network (CNN) and Recurrent Neural Network (RNN) combined with a dynamic attention fusion layer to effectively cope with the problem of noise and unstructured data in a real classroom environment. Through experimental evaluation of over 50,000 labeled samples from 120 students in 10 math courses, the system achieved an F1 score of 87.6%, significantly outperforming both unimodal and traditional fusion approaches. This study provides a scalable framework for emotionally aware educational technology and demonstrates the benefits of multimodal data fusion.