Semi-Supervised Deep Neural Network for Joint Intensity Estimation of Multiple Facial Action Units

Abstract

Facial action units (AUs) are defined to depict movements of facial muscles, which are basic elements to encode facial expressions. Automatic AU intensity estimation is an important task in affective computing. Previous works leverage the representation power of deep neural networks (DNNs) to improve the performance of intensity estimation. However, a large number of intensity annotations are required to train DNNs that contain millions of parameters. But it is expensive and difficult to build a large-scale database with AU intensity annotation since AU annotation requires annotators have strong domain expertise. We propose a novel semi-supervised deep convolutional network that leverages extremely limited AU annotations for AU intensity estimation. It requires only intensity annotations of keyframes of training sequences. Domain knowledge on AUs is leveraged to provide weak supervisory information, including relative appearance similarity, temporal intensity ordering, facial symmetry, and contrastive appearance difference. We also propose a strategy to train a model for joint intensity estimation of multiple AUs under the setting of semi-supervised learning, which greatly improves the efficiency during inference. We perform empirical experiments on two public benchmark expression databases and make comparisons with state-of-the-art methods to demonstrate the effectiveness of the proposed method.