Personality Traits Assessment Through Social Media Likes: A Trimodal Learning Approach Integrating Correlated Video, Audio, and Text

Abstract

With the rise of social media, particularly short video platforms, digital footprints, to some extent, reflect individuals' real-life emotions and can be viewed as an expression of personality. Naturally, users' “likes” on these social platforms, as projections of their character, have become valuable for understanding their personality traits. This paper proposes a trimodal learning-based framework for personality traits assessment, focusing on social media likes. The method integrates video, audio, and text modalities to capture a correlated representation of user personality. For video, a novel parallel temporal-spatial attention fusion mechanism is employed to capture both the temporal dynamics and spatial details of video content. For audio, a multiscale attention spectrogram approach is used, enabling the model to extract hierarchical features at varying temporal resolutions, thereby capturing both high-level trends and subtle tonal variations within audio sequences. For text analysis, the model combines Transformer and BiLSTM layers. Transformers provide powerful context-aware embeddings, which can capture semantic relationships, while BiLSTM block handle sequential dependencies, enhancing the understanding of language patterns often found in personality-relevant text. Finally, the features extracted from each modality are fused through a weighted form, leveraging complementary information across video, audio, and text to produce a comprehensive personality classification. Diverse numerical simulations are provided to verify the performance of our theoretical results compared with mainstream methods.