Learning Natural Consistency Representation for Face Forgery Video Detection

TL;DR

The paper tackles the lack of generalization in face forgery detectors by learning a visual-only Natural Consistency representation (NACO) from real videos. It combines CNN-based frame features with Transformer-based sequence modeling and introduces two self-supervised tasks, Spatial Predictive Module and Temporal Contrastive Module, to enforce natural spatiotemporal coherence. Empirical results show strong cross-dataset and cross-manipulation generalization and robustness, with efficient parameter usage due to freezing the backbone during detection. The approach yields interpretable localization cues and demonstrates potential for broader applications in media forensics beyond forgery detection. The work advances modality-agnostic, self-supervised detection by leveraging high-level temporal consistency in real videos.

Abstract

Face Forgery videos have elicited critical social public concerns and various detectors have been proposed. However, fully-supervised detectors may lead to easily overfitting to specific forgery methods or videos, and existing self-supervised detectors are strict on auxiliary tasks, such as requiring audio or multi-modalities, leading to limited generalization and robustness. In this paper, we examine whether we can address this issue by leveraging visual-only real face videos. To this end, we propose to learn the Natural Consistency representation (NACO) of real face videos in a self-supervised manner, which is inspired by the observation that fake videos struggle to maintain the natural spatiotemporal consistency even under unknown forgery methods and different perturbations. Our NACO first extracts spatial features of each frame by CNNs then integrates them into Transformer to learn the long-range spatiotemporal representation, leveraging the advantages of CNNs and Transformer on local spatial receptive field and long-term memory respectively. Furthermore, a Spatial Predictive Module~(SPM) and a Temporal Contrastive Module~(TCM) are introduced to enhance the natural consistency representation learning. The SPM aims to predict random masked spatial features from spatiotemporal representation, and the TCM regularizes the latent distance of spatiotemporal representation by shuffling the natural order to disturb the consistency, which could both force our NACO more sensitive to the natural spatiotemporal consistency. After the representation learning stage, a MLP head is fine-tuned to perform the usual forgery video classification task. Extensive experiments show that our method outperforms other state-of-the-art competitors with impressive generalization and robustness.

Figures (4)

• Figure 1: (a) Real face videos exhibit natural spatiotemporal consistency while fake videos generated from unknown forgery methods or under different perturbations both show inconsistencies. (b) t-SNE tSNE visualization of our NACO on uncompressed (raw) and heavily compressed (c40) FF++ which includes four different forgery methods.
• Figure 2: The pipeline of our proposed NACO. In the first stage, real videos are first extracted into spatial feature sequence $\mathbf{F}$ by CNN encoder, which is fed into the Transformer to learn long-range spatiotemporal representation $\mathbf{z}$. Further, two designed auxiliary tasks: SPM and TCM are introduced to enhance the natural consistency learning on real face videos. In the second stage, the encoder and Transformer are frozen and a fully-connected classification head (two-layer MLP) guided by learned NACO representation is optimized to perform the usual face forgery video classification task.
• Figure 3: Comparisons on different number of real samples in natural consistency representation learning stage.
• Figure 4: Forgery localization. Grad-CAM results on five consecutive frames on FF++ (c23). We find that our method can effectively respond to the inconsistencies in fake videos and localize the forgery areas.