DeepFake-Adapter: Dual-Level Adapter for DeepFake Detection

TL;DR

This work tackles the generalization gap in deepfake detection by leveraging high-level semantics learned by large Vision Transformers (ViTs). It introduces DeepFake-Adapter, a parameter-efficient dual-level adapter framework consisting of Globally-aware Bottleneck Adapters (GBA) and Locally-aware Spatial Adapters (LSA) that operates with a frozen ViT backbone to fuse global and local forgery cues. Extensive experiments on FaceForensics++, Celeb-DF, DFDC, and DF1.0 demonstrate strong discrimination and, more importantly, cross-dataset and cross-manipulation generalization, outperforming full-tuning and other adapters in many settings. The results suggest that incorporating high-level semantic representations from large ViTs can substantially improve robustness to unseen and degraded deepfakes, with practical benefits due to low additional parameter cost.

Abstract

Existing deepfake detection methods fail to generalize well to unseen or degraded samples, which can be attributed to the over-fitting of low-level forgery patterns. Here we argue that high-level semantics are also indispensable recipes for generalizable forgery detection. Recently, large pre-trained Vision Transformers (ViTs) have shown promising generalization capability. In this paper, we propose the first parameter-efficient tuning approach for deepfake detection, namely DeepFake-Adapter, to effectively and efficiently adapt the generalizable high-level semantics from large pre-trained ViTs to aid deepfake detection. Given large pre-trained models but limited deepfake data, DeepFake-Adapter introduces lightweight yet dedicated dual-level adapter modules to a ViT while keeping the model backbone frozen. Specifically, to guide the adaptation process to be aware of both global and local forgery cues of deepfake data, 1) we not only insert Globally-aware Bottleneck Adapters in parallel to MLP layers of ViT, 2) but also actively cross-attend Locally-aware Spatial Adapters with features from ViT. Unlike existing deepfake detection methods merely focusing on low-level forgery patterns, the forgery detection process of our model can be regularized by generalizable high-level semantics from a pre-trained ViT and adapted by global and local low-level forgeries of deepfake data. Extensive experiments on several standard deepfake detection benchmarks validate the effectiveness of our approach. Notably, DeepFake-Adapter demonstrates a convincing advantage under cross-dataset and cross-manipulation settings. The code has been released at https://github.com/rshaojimmy/DeepFake-Adapter.

Figures (7)

• Figure 1: Example images and distributions of pre-trained Xception and ViT features after linear-probe on (a) DeepFake and (b) FaceSwap splits of FaceForensics++ dataset.
• Figure 2: Overall architecture of proposed model. The model consists of $N$ stages. Each stage contains MHSA and MLP layers of pre-trained ViT, GBA and LSA (LSA-H and LSA-I) of proposed DeepFake-Adapter.
• Figure 3: Details of GBA and LSA of proposed DeepFake-Adapter. (a) Head part and (b) Interaction part of LSA capture local low-level forgeries that interact with features from pre-trained ViT via a series of cross-attention. (c) GBA adapts the pre-trained ViT with global low-level forgeries in a bottleneck structure.
• Figure 4: Comparison of Grad-CAM visualizations between Xception and the proposed model in cross-manipulation evaluation. (Best viewed in color)
• Figure 5: Comparison of Grad-CAM visualizations between Xception and the proposed model in cross-dataset evaluation among DFDC, Celeb-DF and DF1.0 datasets. (Best viewed in color)