Generalized Face Forgery Detection via Adaptive Learning for Pre-trained Vision Transformer

TL;DR

The paper tackles the poor cross-domain generalization of face forgery detectors by freezing pre-trained Vision Transformer (ViT) weights and introducing an adaptive learning framework. It adds a Global Adaptive Module (GAM) to capture global forgery cues, a Local Adaptive Module (LAM) to incorporate local context from CNN features, and a Fine-grained Adaptive Learning (FAL) mechanism to emphasize subtle, fine-grained discrepancies; the training objective blends cross-entropy with a circle-loss-based fine-grained term. Across intra- and cross-dataset tests, including seven unseen datasets and cross-manipulation scenarios, FA-ViT achieves state-of-the-art or near-state-of-the-art performance and exhibits superior robustness to real-world perturbations. Ablation studies confirm that preserving the ViT backbone while applying adaptive modules and fine-grained guidance yields the strongest generalization, supported by extensive analyses of GAM/LAM placements, hyperparameters, and pre-trained initializations. The work provides a practical, generalizable approach to deepfake detection with clear insights into what components most contribute to cross-domain robustness and how to balance global, local, and fine-grained cues.

Abstract

With the rapid progress of generative models, the current challenge in face forgery detection is how to effectively detect realistic manipulated faces from different unseen domains. Though previous studies show that pre-trained Vision Transformer (ViT) based models can achieve some promising results after fully fine-tuning on the Deepfake dataset, their generalization performances are still unsatisfactory. One possible reason is that fully fine-tuned ViT-based models may disrupt the pre-trained features [1, 2] and overfit to some data-specific patterns [3]. To alleviate this issue, we present a \textbf{F}orgery-aware \textbf{A}daptive \textbf{Vi}sion \textbf{T}ransformer (FA-ViT) under the adaptive learning paradigm, where the parameters in the pre-trained ViT are kept fixed while the designed adaptive modules are optimized to capture forgery features. Specifically, a global adaptive module is designed to model long-range interactions among input tokens, which takes advantage of self-attention mechanism to mine global forgery clues. To further explore essential local forgery clues, a local adaptive module is proposed to expose local inconsistencies by enhancing the local contextual association. In addition, we introduce a fine-grained adaptive learning module that emphasizes the common compact representation of genuine faces through relationship learning in fine-grained pairs, driving these proposed adaptive modules to be aware of fine-grained forgery-aware information. Extensive experiments demonstrate that our FA-ViT achieves state-of-the-arts results in the cross-dataset evaluation, and enhances the robustness against unseen perturbations. Particularly, FA-ViT achieves 93.83\% and 78.32\% AUC scores on Celeb-DF and DFDC datasets in the cross-dataset evaluation. The code and trained model have been released at: https://github.com/LoveSiameseCat/FAViT.

Figures (10)

• Figure 1: Overview of the proposed Forgery-aware Adaptive Vision Transformer (FA-ViT). Global Adaptive Modules (GAM) and Local Adaptive Modules (LAM) are inserted into the fixed pre-trained ViT to learn different types of forgery-aware information. Fine-grained Adaptive Learning (FAL) further guides these modules to mine fine-grained clues from subtle discrepancies in fine-grained pairs. We omit CNN blocks for better visualization.
• Figure 2: The details of the proposed global adaptive module.
• Figure 3: Examples of artifacts in manipulated faces that cause local inconsistencies.
• Figure 4: The details of the proposed local adaptive module.
• Figure 5: The local attention map of the query token (marked by a red box) under different combination of $\psi_{1}$ and $\psi_{2}$.