Lip reading from multi view facial images using 3D-AAM

Abstract

Lip reading is a technique to recognize the spoken words base on lip movement. In this process, it is important to detect the correct features of the facial images. However, detection is not easy in the real situations because the facial images may be taken from various angles. To cope with this problem, lip reading from multi view facial images has been conducted in several research institutes. In this paper, we propose a lip reading approach using the 3D Active Appearance Models (AAM) features and the Hidden Markov Model (HMM)-based recognition model. The AAM is a parametric model constructed from both shape and appearance parameters. The parameters are compressed into the combination parameters in the AAM, and are used in lip reading or some other facial image processing applications. The 3D-AAM extends the traditional 2D shape model to 3D shape model built from three different view angles (frontal, left, and right profile). It provides an effective algorithm to align the model with the RGB and the 3D range images obtained by the RGBD-camera. The benefit of using 3D-AAM in lip reading is that it enables to recognize the spoken words from any angle of the facial images. In the experiment, we compared the accuracy of lip reading using 3D-AAM with that of the traditional 2D-AAM on various angles of facial images. Based on the result, we confirmed that 3D-AAM is effective in cross view lip reading despite using only the frontal images in the HMM training phase.