Template-based automatic high-speed relighting of faces

Abstract

This paper presents a framework which estimates the surface normals of human face, surface albedo using an average 3D face template and subsequently replaces the original lighting on the face with a novel lighting in real time. The system uses a facial feature tracking algorithm, which locates and estimates the orientation of face. The 2D facial landmarks thus obtained are used to morph the template model to resemble the input face. The lighting conditions are approximated as a linear combination of Spherical Harmonic bases. A photometric refinement is applied to accurately estimate the surface normal and thus surface albedo. A novel skin-mask construction algorithm is also used to restrict the processing to facial region of the input image. The face is relighted with novel illumination parameters. A novel, cost-effective, seamless blending operation is performed to achieve efficacious and realistic outputs. The process is fully automatic and is executed in real time.