Multimodal Emotional Detection System for Virtual Educational Environments: Integration Into Microsoft Teams to Improve Student Engagement

Abstract

In modern educational settings, especially virtual environments, understanding students' emotional states has become crucial in improving participation, engagement, and academic performance. Current emotional detection systems, such as Affectiva and Emotient, offer facial and vocal emotional recognition but present limitations regarding integration, scalability, and adaptability to widely used platforms such as Microsoft Teams. These systems often require complex integrations and are not readily adaptable to the dynamic nature of the educational environment. This study proposes an innovative solution for real-time emotional detection, using a multimodal approach that combines speech and facial expression analysis within the Microsoft Teams platform. By integrating TensorFlow, PyTorch, OpenCV, and Dlib, we developed a system capable of accurately detecting emotions such as happiness, stress, and calm, with up to 95% detection precision for positive emotions. The system was tested in an educational environment, demonstrating its ability to process multiple interactions simultaneously without compromising performance. Results include high precision in essential emotion detection and significant improvements in student engagement. However, the system showed limitations in detecting more complex emotions such as stress and frustration, suggesting further refining the model.