Towards understanding perceptual diferences between genuine and face-swapped videos

Abstract

In this paper, we report on perceptual experiments indicating that there are distinct and quantitatively measurable diferences in the way we visually perceive genuine versus face-swapped videos. Recent progress in deep learning has made face-swapping techniques a powerful tool for creative purposes, but also a means for unethical forgeries. Currently, it remains unclear why people are misled, and which indicators they use to recognize potential manipulations. Here, we conduct three perceptual experiments focusing on a wide range of aspects: the conspicuousness of artifacts, the viewing behavior using eye tracking, the recognition accuracy for diferent video lengths, and the assessment of emotions. Our experiments show that responses difer distinctly when watching manipulated as opposed to original faces, from which we derive perceptual cues to recognize face swaps. By investigating physiologically measurable signals, our fndings yield valuable insights that may also be useful for advanced algorithmic detection.