Separable Facial Image Editing via Warping with Displacement Vectors

Abstract

Recent deep learning techniques have enabled simple methods of image editing through textual instructions; however, it remains challenging to perform specific and quantitative editing. In contrast, intuitive image editing methods that use displacement vectors often require the preservation of the original appearance after editing. This study proposes a new method for precisely editing expressions and orientations in facial images. Existing methods that rely on displacement vectors are unable to independently edit facial expressions and orientations. Our method offers separate editing capabilities by introducing multiple displacement vectors with two roles, which are used for transferring three-dimensional (3D) keypoints. These keypoints are then used in image warping to achieve stable deformations. The process involves extracting the movements of 3D keypoints with the images sampled from movies, which are then used to train a deep neural network for expression edits. In addition, the 3D rotation matrix for the keypoints is calculated to handle the change of face orientation. The edited keypoints are passed through a thin-plate spline motion model to warp an input facial image used as an identity. Our method outperforms existing facial image editing and warping methods in preserving face identity by 3.7% in terms of the structural similarity, as demonstrated by quantitative comparisons, and produces more natural results qualitatively.