SG-TE: Spatial Guidance and Temporal Enhancement Network for Facial-Bodily Emotion Recognition

Abstract

To overcome the deficiencies of single-modal emotion recognition based on facial expression or bodily posture in natural scenes, a spatial guidance and temporal enhancement (SG-TE) network is proposed for facial-bodily emotion recognition. First, ResNet50, DNN and spatial ransformer models are used to capture facial texture vectors, bodily skeleton vectors and whole-body geometric vectors, and an intraframe correlation attention guidance (S-CAG) mechanism, which guides the facial texture vector and the bodily skeleton vector by the whole-body geometric vector, is designed to exploit the spatial potential emotional correlation between face and posture. Second, an interframe significant segment enhancement (T-SSE) structure is embedded into a temporal transformer to enhance high emotional intensity frame information and avoid emotional asynchrony. Finally, an adaptive weight assignment (M-AWA) strategy is constructed to realise facial-bodily fusion. The experimental results on the BabyRobot Emotion Dataset (BRED) and Context-Aware Emotion Recognition (CAER) dataset indicate that the proposed network reaches accuracies of 81.61% and 89.39%, which are 9.61% and 9.46% higher than those of the baseline network, respectively. Compared with the state-of-the-art methods, the proposed method achieves 7.73% and 20.57% higher accuracy than single-modal methods based on facial expression or bodily posture, respectively, and 2.16% higher accuracy than the dual-modal methods based on facial-bodily fusion. Therefore, the proposed method, which adaptively fuses the complementary information of face and posture, improves the quality of emotion recognition in real-world scenarios.