Emotion Recognition Based on a EEG–fNIRS Hybrid Brain Network in the Source Space

Abstract

Background/Objectives: Studies have shown that emotion recognition based on electroencephalogram (EEG) and functional near-infrared spectroscopy (fNIRS) multimodal physiological signals exhibits superior performance compared to that of unimodal approaches. Nonetheless, there remains a paucity of in-depth investigations analyzing the inherent relationship between EEG and fNIRS and constructing brain networks to improve the performance of emotion recognition. Methods: In this study, we introduce an innovative method to construct hybrid brain networks in the source space based on simultaneous EEG-fNIRS signals for emotion recognition. Specifically, we perform source localization on EEG signals to derive the EEG source signals. Subsequently, causal brain networks are established in the source space by analyzing the Granger causality between the EEG source signals, while coupled brain networks in the source space are formed by assessing the coupling strength between the EEG source signals and the fNIRS signals. The resultant causal brain networks and coupled brain networks are integrated to create hybrid brain networks in the source space, which serve as features for emotion recognition. Results: The effectiveness of our proposed method is validated on multiple emotion datasets. The experimental results indicate that the recognition performance of our approach significantly surpasses that of the baseline method. Conclusions: This work offers a novel perspective on the fusion of EEG and fNIRS signals in an emotion-evoked experimental paradigm and provides a feasible solution for enhancing emotion recognition performance.