May 6, 2009 9:35 WSPC/115-IJPRAI SPI-J068 00724
International Journal of Pattern Recognition
and Artificial Intelligence
Vol. 23, No. 3 (2009) 379–400
c
© World Scientific Publishing Company
TOWARDS PRACTICAL FACIAL
FEATURE DETECTION
MICAH ECKHARDT
Machine Perception Laboratory
Institute for Neural Computation
University of California, San Diego
La Jolla, CA 92093, USA
micahrye@mplab.ucsd.edu
IAN FASEL
Department of Computer Science
University of Arizona
Tucson, AZ 85721, USA
ianfasel@cs.arizona.edu
JAVIER MOVELLAN
Machine Perception Laboratory
Institute for Neural Computation
University of California, San Diego
La Jolla, CA 92093, USA
movellan@mplab.ucsd.edu
Localizing facial features is a critical component in many computer vision applications
such as expression recognition, face recognition, face tracking, animation, and red-eye
correction. Practical applications require detectors that operate reliably under a wide
range of conditions, including variations in illumination, pose, ethnicity, gender and age.
One challenge for the development of such detectors is the inherent trade-off between
robustness and precision. Robust detectors tend to provide poor localization and detec-
tors sensitive to small changes in local structure, which are needed for precise local-
ization, generate a large number of false alarms. Here we present an approach to this
trade-off based on context dependent inference. First, robust detectors are used to detect
contexts in which target features occur, then precise detectors are trained to localize the
features given the detected context. This paper describes the approach and presents a
thorough empirical examination of the parameters needed to achieve practical levels of
performance, including the size of the training database, size of the detector’s receptive
fields and methods for information integration. The approach operates in real time and
achieves, to our knowledge, the most accurate localization performance to date.
Keywords: Machine vision; feature detection; image registration.
379

May 6, 2009 9:35 WSPC/115-IJPRAI SPI-J068 00724
380 M. Eckhardt, I. Fasel & J. Movellan
1. Introduction
Localizing facial features is a critical component in many computer vision
applications, including expression recognition, avatar animation, face recogni-
tion, head pose estimation, and artifact removal (e.g. red eye effect) in digital
camera.5, 8, 9, 15, 25, 27, 30 Despite its importance, facial feature localization is still an
unsolved problem for applications that need to operate under a wide range of con-
ditions that include realistic variations in illumination, ethnicity, gender, age, pose,
and imaging hardware.5
While many approaches to feature detection have been proposed and tested on
several benchmark datasets, the particular details for how any of these methods
could be pushed to performance levels reliable enough for practical use is rarely
studied systematically. The challenges are both theoretical and empirical. One the-
oretical challenge is an inherent trade-off between robustness and precision. Robust
detectors that work reliably in a wide variety of conditions tend to provide poor
localization performance, while detectors capable of distinguishing small deviations
from target locations tend to generate a large number of false alarms. Here we
present an approach to this trade-off based on context dependent inference (CDI):
first, robust detectors are trained to detect the context in which target features
occur, and then precise detectors are trained to localize the target features given
the context.
Another challenge is the historical aversion of the computer vision community
for empirical parametric studies. Such studies were critical in the development of
fields such as automatic speech recognition,16 but their importance has in general
not been recognized yet in the computer vision community. The consequence has
been slower technological progress, scientific progress, and practical application of
computer vision in many domains.
Here we document the process of developing state of the art feature detectors
using machine learning methods under the CDI approach described above. Empir-
ical studies are presented on a large and challenging dataset of faces obtained from
the Web, including a wide variety of illumination and rendering conditions, both
indoors and outdoors. We investigatge and describe the effect on performance of a
wide range of intervening factors. The parameters and algorithms that optimized
performance, as well as those that did not, are clearly described to facilitate repli-
cability and progress in the field.
2. Overview of the Approach
There is a fundamental trade-off inherent to the problem of feature localization.
While robust feature detectors tend to localize poorly, detectors sensitive to small
variations, which are needed for precise localization, tend to produce a large number
of false alarms. One common approach to solve this trade-off is based on the opera-
tion of a set of independent feature detectors.5, 6, 10, 14 The output of these detectors
(e.g. a detector for each eye, a detector for the tip of the nose, a detector for points