Multimodal emotion recognition using facial expressions, body gestures, speech, and text modalities

Abstract

Automatic emotion recognition from multimodal content has become an important and growing research field in human-computer interaction. Recent literature has used either audio or facial expression for emotion detection. However, emotion and body gestures are closely related to one another. This paper explores the effectiveness of using text, audio, facial expression and body gesture modalities of multimodal content and machine learning and deep learning based models for building more accurate and robust automatic multimodal emotion recognition systems. First, we get the best accuracy from the individual modalities. Then we use feature level fusion and ensemble based decision level fusion to combine multiple modalities to get better results. Proposed models were tested on IEMOCAP dataset and results show that proposed models with multiple modalities are more accurate compared to unimodal models in classifying emotions.