Detecting Deepfakes Using Two-Stream Framework Based on EfficientNet and Vision Transformer

Abstract

The ever-growing threat of deepfake technology has stimulated research in deepfake forensics to identify the genuineness of media materials on the Internet. The majority of existing techniques employ the convolutional neural network (CNN) framework to identify deepfake by capturing the image’s local features. However, due to the limited scope of receptive fields, relying solely on the feature extraction capabilities of CNNs is insufficient to meet the detection demands posed by increasingly diverse forged samples in real-world scenarios. To address these challenges, this study introduces an innovative model for deepfake identification, leveraging the strengths of EfficientNet and vision transformers to integrate local and global features. This integration allows for more comprehensive learning of deep image features, thereby enhancing detection accuracy. To further enhance detection capabilities and bolster the resilience of our approach against image compression, we designed a dual-stream architecture that incorporates noise features extracted from steganalysis rich model filters as an additional input. By integrating multiple information streams, the model’s performance is effectively improved. From the experiments, the proposed model achieved high accuracy rates of 0.845, 0.999, and 0.815 on the video test sets FaceForensics, Celeb-DF-V2, and Wild Data, respectively. It also attained the best results on the photo test sets FaceApp, FaceSwapApp, and FaceFuse, with scores of 0.998, 0.925, and 0.918. Notably, the model demonstrated strong performance across various deepfake datasets, especially those sourced from social networks. With 88M parameters compared to 101M in the next-best competing method, the proposed model confirms its effectiveness in real-world case studies while maintaining reduced computational complexity.