Facial Expression Recognition Method Based on a Part-Based Temporal Convolutional Network with a Graph-Structured Representation

Abstract

Facial expressions are controlled by facial muscles and can be regarded as appearance and shape variations in key parts. A key challenge in facial expression recognition is capturing effective information from a facial image. In this paper, we propose a basic graph contour that is based on key parts for facial expression recognition. Each node on the graph contour represents a landmark, and each edge represents the connection between the two selected nodes. To further investigate the graph representation and to make the graphs more distinctive, we use a Gabor filter to extract appearance variations around the graph nodes while applying an affine transformation to capture the shape variations from graphs without expression in graphs with expression. Then, to serve as an efficient network for processing in which the graph extracts the appearance and shape representations, we introduce the temporal convolutional network (TCN). Finally, we propose a part-based temporal convolutional network (PTCN) that emphasizes the key facial parts. The experimental results demonstrate that this method realizes significant improvements over state-of-the-art methods utilizing three widely used facial databases: Oulu-CASIA, CK+, and MMI.