Computer Vision and Pattern Recognition 2005
Recognizing Facial Expression: Machine Learning and Application to
Spontaneous Behavior
Marian Stewart Bartlett1, Gwen Littlewort1, Mark Frank2, Claudia Lainscsek1,
Ian Fasel1, Javier Movellan1
Institute for Neural Computation, University of California, San Diego
2 Rutgers University, New Brunswick, NJ
mbartlett@ucsd.edu
Abstract
We present a systematic comparison of machine learning
methods applied to the problem of fully automatic recogni-
tion of facial expressions. We report results on a series of
experiments comparing recognition engines, including Ad-
aBoost, support vector machines, linear discriminant anal-
ysis. We also explored feature selection techniques, includ-
ing the use of AdaBoost for feature selection prior to clas-
sification by SVM or LDA. Best results were obtained by
selecting a subset of Gabor filters using AdaBoost followed
by classification with Support Vector Machines. The system
operates in real-time, and obtained 93% correct general-
ization to novel subjects for a 7-way forced choice on the
Cohn-Kanade expression dataset. The outputs of the clas-
sifiers change smoothly as a function of time and thus can
be used to measure facial expression dynamics. We applied
the system to to fully automated recognition of facial ac-
tions (FACS). The present system classifies 17 action units,
whether they occur singly or in combination with other ac-
tions, with a mean accuracy of 94.8%. We present prelimi-
nary results for applying this system to spontaneous facial
expressions.
1 Introduction
We present results on a user independent fully automatic
system for real time recognition of basic emotional expres-
sions from video. The system automatically detects frontal
faces in the video stream and codes each frame with respect
to 7 dimensions: Neutral, anger, disgust, fear, joy, sadness,
surprise. A second version of the system detects 17 action
units of the Facial Action Coding System (FACS). We con-
ducted empirical investigations of machine learning meth-
ods applied to this problem, including comparison of recog-
nition engines and feature selection techniques. Best results
were obtained by selecting a subset of Gabor filters using
AdaBoost and then training Support Vector Machines on
the outputs of the filters selected by AdaBoost. The com-
bination of AdaBoost and SVM’s enhanced both speed and
accuracy of the system. The system presented here is fully
automatic and operates in real-time. We present prelimi-
nary results for recognizing spontaneous expressions in an
interview setting.
2 Facial Expression Data
The facial expression system was trained and tested on
Cohn and Kanade’s DFAT-504 dataset [7]. This dataset con-
sists of 100 university students ranging in age from 18 to
30 years. 65% were female, 15% were African-American,
and 3% were Asian or Latino. Videos were recoded in ana-
log S-video using a camera located directly in front of the
subject. Subjects were instructed by an experimenter to per-
form a series of 23 facial expressions. Subjects began each
display with a neutral face. Before performing each display,
an experimenter described and modeled the desired display.
Image sequences from neutral to target display were digi-
tized into 640 by 480 pixel arrays with 8-bit precision for
grayscale values. For our study, we selected the 313 se-
quences from the dataset that were labeled as one of the 6
basic emotions. The sequences came from 90 subjects, with
1 to 6 emotions per subject. The first and last frames (neu-
tral and peak) were used as training images and for testing
generalization to new subjects, for a total of 626 examples.
The trained classifiers were later applied to the entire se-
quence.
2.1 Real-time Face Detection
We developed a real-time face detection system that em-
ploys boosting techniques in a generative framework [5]
and extends work by [17]. Enhancements to [17] include
employing Gentleboost instead of Adaboost, smart feature
search, and a novel cascade training procedure, combined
in a generative framework. Source code for the face detec-
tor is freely available at http://kolmogorov.sourceforge.net.