FEEL: fast and effective emotion labeling, a dual ensemble approach for effective facial emotion recognition

Abstract

Facial expressions are a vital channel for communicating emotions and personality traits, making automatic emotion recognition from facial images a task of growing importance with wide-ranging applications. While deep learning models have shown considerable promise in this domain, most existing approaches are unimodal and limited to classifying only six basic emotions. This study introduces a dual-ensemble deep learning framework designed to recognize both basic and complex blended emotions with high accuracy. The first ensemble focuses on detecting primary emotions using DenseNet-169, VGG-16, and ResNet-50 as base models. The second ensemble focuses on identifying nuanced emotional blends, utilizing Xception and vision transformer (ViT) architectures. A squeeze-and-excitation (SE) block is incorporated to emphasize the most salient features, thereby enhancing overall model performance. The proposed framework is trained and evaluated on the widely used Facial Expression Recognition (FER)2013 and Indonesian Mixed Emotion Dataset (IMED) datasets. Experimental results demonstrate its effectiveness, achieving 95% accuracy for basic emotion recognition and 88% for blended emotions. These findings underscore the potential of the proposed approach to advance human-computer interaction by improving both the accuracy and depth of automated emotion recognition systems.