Fully Automatic Facial Action Recognition in Spontaneous Behavior
Marian Stewart Bartlett1, Gwen Littlewort1, Mark Frank2, Claudia Lainscsek1,
Ian Fasel1, Javier Movellan1
1Institute for Neural Computation, University of California, San Diego
2 SUNY Buffalo, New York
mbartlett@ucsd.edu
Abstract
We present results on a user independent fully automatic
system for real time recognition of facial actions from the
Facial Action Coding System (FACS). The system automat-
ically detects frontal faces in the video stream and codes
each frame with respect to 20 Action units. We present
preliminary results on a task of facial action detection in
spontaneous expressions during discourse. Support vector
machines and AdaBoost classifiers are compared. For both
classifiers, the output margin predicts action unit intensity.
1 Introduction
In order to objectively capture the richness and complex-
ity of facial expressions, behavioral scientists have found it
necessary to develop objective coding standards. The facial
action coding system (FACS) [2] is the most objective and
comprehensive coding system in the behavioral sciences. A
human coder decomposes facial expressions in terms of 46
component movements, which roughly correspond to the
individual facial muscles. An example is shown in Figure 1.
Several research groups have recognized the importance of
automatically recognizing FACS [1, 9, 8, 5]. Here we de-
scribe progress on a system for fully automated facial action
coding.
We present results on a user independent fully automatic
system for real time recognition of facial actions from the
Facial Action Coding System (FACS). The system automat-
ically detects frontal faces in the video stream and codes
each frame with respect to 20 Action units. In previous
work, we conducted empirical investigations of machine
learning methods applied to the related problem of classi-
fying expressions of basic emotions [6]. We compared Ad-
aBoost, support vector machines, and linear discriminant
analysis, as well as feature selection methods techniques.
Best results were obtained by selecting a subset of Gabor
filters using AdaBoost and then training Support Vector Ma-
chines on the outputs of the filters selected by AdaBoost.
Figure 1. Example FACS codes for a prototypical expres-
sion of fear. Spontaneous expressions may contain only a
subset of these Action Units.
The combination of AdaBoost and SVM’s enhanced both
speed and accuracy of the system. An overview of the sys-
tem is shown in Figure 2. Here we apply this system to the
problem of detecting facial actions in spontaneous expres-
sions. The system presented here detects 20 action units, is
fully automatic, and operates in real-time.
2 Automated System
2.1 Real-time Face Detection
We developed a real-time face detection system that em-
ploys boosting techniques in a generative framework [3]
and extends work by [10]. Enhancements to [10] include
employing Gentleboost instead of AdaBoost, smart feature
search, and a novel cascade training procedure, combined
in a generative framework. Source code for the face detec-
tor is freely available at http://kolmogorov.sourceforge.net.
Accuracy on the CMU-MIT dataset, a standard public data
set for benchmarking frontal face detection systems, is 90%
detections and 1/million false alarms, which is state-of-the-
art accuracy. The CMU test set has unconstrained lighting
and background. With controlled lighting and background,
such as the facial expression data employed here, detection