Temporal Attentive Adversarial Domain Adaption for Cross Cultural Affect Recognition

Abstract

Continuous affect recognition is becoming an increasingly attractive research topic, recent works mainly focus on modeling the temporal dependency and multi-modal fusion to boost the performance. Despite recent improvement, the cross-cultural affect recognition in videos is still not well-explored. In this paper, we propose the temporal attentive adversarial domain adaption for cross cultural affect recognition. The LSTM is firstly used to encode the contextual representation for each frame. Then, a DNN based regressor is used to estimate the affective dimension arousal or valence, and optimized to promote the encoded representation is emotion discriminative. In addition, another DNN based sequence level culture classifier, which takes the fused representation of each frame as the input, is used to recognize the culture of the input sequence, and optimized to encourage the encoded representation is culture invariant. Since different frames over a video may contribute not equally in recognizing the culture, we propose to add another frame level culture classifier, which could adaptively and attentively assign more weighting scores for the important frames for recognizing the culture. The proposed method is evaluated on the benchmark dataset AVEC2019 CES. Our experimental results show that the proposed method improves the performance compared to state-of-the-art methods, with the concordance correlation coefficient (CCC) reaching 0.576 for arousal and 0.472 for valence, on the cross cultural test set.