Real-time face pose tracking and facial expression synthesizing for the animation of 3D avatar

Abstract

This paper introduces a novel approach for vision-based head motion tracking and facial expression cloning to create the realistic facial animation of 3D avatar in real time. The exact head pose estimation and facial expression tracking are critical problems to be solved in developing a vision based computer animation. The proposed method consists of dynamic head pose estimation and facial expression cloning. The proposed head pose estimation technique can robustly estimate 3D head pose from a sequence of input video images. Given an initial reference template of head image and corresponding 3D head pose, full the head motion is recovered by projecting a cylindrical head model to the face image. By updating the template dynamically, it is possible to recover head pose robustly regardless of light variation and self-occlusion. In addition, to produce a realistic 3D face animation, the variation of major facial feature points is tracked by use of optical flow and retargeted to the 3D avatar. We exploit Gaussian RBF to deform the local region of 3D face model around the major feature points. During the model deformation, the clusters of the regional feature points around the major facial features are estimated and the positions of the clusters are changed according to the variation of the major feature points. From the experiments, we can prove that the proposed vision-based animation technique efficiently estimate 3D head pose and produce realistic 3D facial animation rather than using feature-based tracking method. © Springer-Verlag Berlin Heidelberg 2007.