Identifying Gender Differences in Multimodal Emotion Recognition Using Bimodal Deep AutoEncoder

Abstract

This paper mainly focuses on investigating the differences between males and females in emotion recognition using electroencephalography (EEG) and eye movement data. Four basic emotions are considered, namely happy, sad, fearful and neutral. The Bimodal Deep AutoEncoder (BDAE) and the fuzzy-integral-based method are applied to fuse EEG and eye movement data. Our experimental results indicate that gender differences do exist in neural patterns for emotion recognition; eye movement data is not as good as EEG data for examining gender differences in emotion recognition; the activation of the brains for females is generally lower than that for males in most bands and brain areas especially for fearful emotions. According to the confusion matrix, we observe that the fearful emotion is more diverse among women compared with men, and men behave more diversely on the sad emotion compared with women. Additionally, individual differences in fear are more pronounced than other three emotions for females.