Large-scale datasets for facial tampering detection with inpainting techniques

Abstract

Objective DeepFake technology，born with the continuous maturation of deep learning techniques，primarily utilizes neural networks to create non-realistic faces. This method has enriched people’s lives as computer vision advances and deep learning technologies mature. It has revolutionized the film industry by generating astonishing visuals and reduc-ing production costs. Similarly，in the gaming industry，it has facilitated the creation of smooth and realistic animation effects. However，the malicious use of image manipulation to spread false information poses significant risks to society，casting doubt on the authenticity of digital content in visual media. Forgery techniques encompass four main categories：face reenactment，face replacement，face editing，and face synthesis. Face editing，a commonly employed image manipu-lation method，involves falsifying facial features by modifying the information related to the five facial regions. As one of the commonly employed methods in facial editing，image inpainting technology involves utilizing known content from an image to fill in missing areas，aiming to restore the image in a way that aligns as closely as possible with human perception. In the context of facial forgery，image inpainting is primarily used for identity falsification，wherein facial features are altered to achieve the goal of replacing a face. The use of image inpainting for facial manipulation similarly introduces sig-nificant disruption to people’s lives. To support research on detection methods for such manipulations，this paper produced a large-scale dataset for face manipulation detection based on inpainting techniques. Method This paper specifically focuses on the field of image tampering detection，utilizing two classic datasets：the high-quality CelebA-HQ dataset，com-prising 25 000 high-resolution（1 024 × 1 024 pixels）celebrity face images，and the low-quality FF++ dataset，consisting of 15 000 face images extracted from video frames. On the basis of the two datasets，facial feature regions（eyebrows，eyes，nose，mouth，and the entire facial area）are segmented using image segmentation methods. Corresponding mask images are created，and the segmented facial regions are directly obscured on the original image. Two deep neural network-based inpainting methods（image inpainting via conditional texture and structure dual generation（CTSDG）and recurrent feature reasoning for image inpainting（RFR））along with a traditional inpainting method（struct completion（SC））were employed. The deep neural network methods require the provision of mask images to indicate the areas for inpainting，while the traditional method could directly perform inpainting on segmented facial feature images. The facial regions were inpainted using these three methods，resulting in a large-scale dataset comprising 600 000 images. This extensive dataset incorporates diverse pre-processing techniques，various inpainting methods，and includes images with different qualities and inpainted facial regions. It serves as a valuable resource for training and testing in related detection tasks，offering a rich dataset for subsequent research in the field，and also establishes a meaningful benchmark dataset for future studies in the domain of face tampering detection. Result We present comparative experiments conducted on the generated dataset，revealing notable findings. Experimental results indicate a 15% decrease in detection accuracy for images derived from the FF++ dataset under the ResNet-50 benchmark detection network. Under the Xception-Net network，the detection accuracy experiences a 5% decline. Furthermore，significant variations in detection accuracy are observed among different facial regions，with the lowest accuracy recorded in the eye region at 0. 91. Generalization experiments suggest that inpainted images from the same source dataset exhibit a certain degree of generalization across different facial regions. In contrast，minimal generalization is observed among datasets created from different source data. Consequently，this dataset also serves as valuable research data for studying the generalization of inpainted images across different facial regions. Visualiza-tion tools demonstrate that the detection network indeed focuses on the inpainted facial features，affirming its attention to the manipulated facial regions. This work provides new research perspectives for methods of detecting image restoration-based manipulations. Conclusion The use of image inpainting techniques for tampering introduces a challenging scenario that can deceive conventional tampering detectors to a certain extent. Researching detection methods for this type of tamper-ing is of practical significance. The provided large-scale face tampering dataset，based on inpainting techniques，encom-passes high- and low-quality images，employing three distinct inpainting methods and targeting various facial features. This dataset offers a novel source of data for research in this field，enhancing diversity and providing benchmark data for further exploration of image restoration-related forgeries. With the scarcity of relevant datasets in this domain，we propose the utili-zation of this dataset as a benchmark for the field of image inpainting tampering detection. This dataset not only supports research in detection methodologies but also contributes to studies on the generalization of such methods. It serves as a foundational resource，filling the gap in the available datasets and facilitating advancements in the detection and generaliza-tion studies in the domain of image inpainting tampering. This benchmark includes a large-scale inpainting image dataset，totaling 600 000 images. The dataset’s quality is evaluated based on accuracy on manipulation detection networks，gener-alizability across different inpainting networks and facial regions，and modules such as data visualization.