A Survey of Facial Capture for Virtual Reality

Abstract

The holograms in Star Wars have inspired many researchers to capture the whole human body in real-time and present it as an avatar into Virtual Reality (VR). Facial capture is important to achieve this because facial expressions are essential for social interaction. However, facial capture works well only when the face is not occluded. While a VR headset has been widely used as a display device for VR, it occludes half of the face and becomes an obstacle. This paper presents a systematic literature review on facial capture for VR. The survey aims to: Review the current state-of-the-art facial capture technologies for VR; identify the types of technologies and context of the use, the methodologies, and theoretical groundings; identify research gaps in facial capture for VR; and identify solutions of facial capture for VR headsets. A realism index is defined to evaluate and compare the collected papers. The results show several technology trends in the facial capture for VR: Tracking facial motion with markers, facial capture in headsets using cameras or sensors, facial performance capture, and hologram/volumetric capture. It is shown that the Modular Codec Avatar is the best facial capture method in a VR headset, whereas Metahuman has the best output effects. This paper also proposes that an open-design VR headset is an effective approach to lower the Total Cost of Ownership (TCO).