A Key Skeleton Points Guided Classroom Action Recognition Method Based on Multimodal Symmetry Fusion

Abstract

Recently, there has been growing interest in utilizing skeleton data for human action recognition due to its compact size and ability to capture action characteristics effectively. However, in complex classroom scenarios, student actions encounter challenges such as high inter-class similarity, differentiation difficulty, and redundancy, which hinder effective differentiation using existing unidirectional feature splicing multimodal methods. Therefore, we propose a key skeleton points guided classroom action recognition method based on multimodal symmetry fusion. This method is primarily characterized by several innovations. Firstly, we utilize a method called Variable Series Mean to select the most significant key skeleton points of actions. Then, these points are input into a model to learn the relevant weight values, guiding the generation of salient regions in RGB images. Finally, in the data fusion stage, we utilize the Symmetric Multi-Modal optimization function to integrate the three data streams, addressing bias issues arising from unidirectional feature splicing methods. We conducted comprehensive experiments on two datasets: NTU 60 and Classroom. Synthesizing results of multiple methods, our method achieves state-of-the-art performance on the NTU 60 dataset and the second-best performance on the private Classroom dataset. Despite not attaining the highest recognition accuracy on the Classroom dataset, this approach offers substantial benefits in terms of time and storage, providing a real-time solution for recognizing student actions in the classroom. Therefore, our method effectively captures and integrates the representation information from different modalities, enabling accurate recognition of student actions in the classroom.