A Novel Multi-Feature Joint Learning Ensemble Framework for Multi-Label Facial Expression Recognition

Abstract

The facial expression is one of the most common ways to reflect human emotions. And understand different classes of facial expressions is an important method in analyzing human perceived and affective states. In the past few decades, facial expression analysis (FEA) has been extensively studied. It illustrates few of the facial expressions are exactly individual of the predefined affective states but are blends of several basic expressions. Some researchers have realized that facial expression recognition can be treated as a multi-label task, but they are still troubled by the inaccurate recognition of multi-label expressions. To overcome this challenge, a novel multi-feature joint learning ensemble framework, called MF-JLE framework, is proposed. The proposed framework combines global features with several different local key features to consider the multiple labels of expressions embodied in many facial action units. The ensemble learning is introduced into the framework, combines the global module and the local module on the loss, and carries out the joint iterative optimization. The ensemble of the whole framework improves the accuracy of multi-label recognition of different modules as weak classifiers. In addition, the traditional multi-classifier cross-entropy loss has been replaced by the binary cross-entropy loss for a better ensemble. The proposed framework is evaluated on the real-world affective faces (RAF-ML) dataset. The experimental results show that the proposed model is better than other methods in both quantitative and qualitative aspects, whether compared with traditional shallow learning methods or recent deep learning methods.