Cross-modal BERT model for enhanced multimodal sentiment analysis in psychological social networks

Abstract

Background: Human emotions in psychological social networks often involve complex interactions across multiple modalities. Information derived from various channels can synergistically complement one another, leading to a more nuanced depiction of an individual’s emotional landscape. Multimodal sentiment analysis emerges as a potent tool to process this diverse array of content, facilitating efficient amalgamation of emotions and quantification of emotional intensity. Methods: This paper proposes a cross-modal BERT model and a cross-modal psychological-emotional fusion (CPEF) model for sentiment analysis, integrating visual, audio, and textual modalities. The model initially processes images and audio through dedicated sub-networks for feature extraction and reduction. These features are then passed through the Masked Multimodal Attention (MMA) module, which amalgamates image and audio features via self-attention, yielding a bimodal attention matrix. Subsequently, textual information is fed into the MMA module, undergoing feature extraction through a pre-trained BERT model. The textual information is then fused with the bimodal attention matrix via the pre-trained BERT model, facilitating emotional fusion across modalities. Results: The experimental results on the CMU-MOSEI dataset showcase the effectiveness of the proposed CPEF model, outperforming comparative models, achieving an impressive accuracy rate of 83.9% and F1 Score of 84.1%, notably improving the quantification of negative, neutral, and positive affective energy. Conclusions: Such advancements contribute to the precise detection of mental health status and the cultivation of a positive and sustainable social network environment.