Scalable Face Security Vision Foundation Model for Deepfake, Diffusion, and Spoofing Detection

TL;DR

FS-VFM addresses the lack of universal, generalizable facial representations for face security tasks by integrating two self-supervised objectives—masked image modeling (MIM) and instance discrimination (ID)—through a 3C framework: intra-region Consistency, inter-region Coherency, and local-to-global Correspondence. It introduces CRFR-P masking to steer MIM toward meaningful intra-region textures and cross-region relations, and couples MIM with an ID branch via a robust self-distillation scheme that uses a full facial view as the target. The backbone is a vanilla Vision Transformer (ViT), enabling scalable growth and straightforward fine-tuning, while FS-Adapter provides an ultra-efficient, plug-and-play transfer mechanism with a Real-Anchor Contrastive Loss to adapt to downstream face security tasks with minimal parameters. Across 11 benchmarks for cross-dataset deepfake detection, cross-domain face anti-spoofing, and unseen diffusion-based forensics, FS-VFM shows superior generalization to unseen manipulations and robust cross-task performance, and FS-Adapter achieves strong efficiency-performance trade-offs. This work advances toward a practical, full-stack face security foundation model with scalable pre-training, adaptable deployment, and broad applicability in authenticating facial content.

Abstract

With abundant, unlabeled real faces, how can we learn robust and transferable facial representations to boost generalization across various face security tasks? We make the first attempt and propose FS-VFM, a scalable self-supervised pre-training framework, to learn fundamental representations of real face images. We introduce three learning objectives, namely 3C, that synergize masked image modeling (MIM) and instance discrimination (ID), empowering FS-VFM to encode both local patterns and global semantics of real faces. Specifically, we formulate various facial masking strategies for MIM and devise a simple yet effective CRFR-P masking, which explicitly prompts the model to pursue meaningful intra-region Consistency and challenging inter-region Coherency. We present a reliable self-distillation mechanism that seamlessly couples MIM with ID to establish underlying local-to-global Correspondence. After pre-training, vanilla vision transformers (ViTs) serve as universal Vision Foundation Models for downstream Face Security tasks: cross-dataset deepfake detection, cross-domain face anti-spoofing, and unseen diffusion facial forensics. To efficiently transfer the pre-trained FS-VFM, we further propose FS-Adapter, a lightweight plug-and-play bottleneck atop the frozen backbone with a novel real-anchor contrastive objective. Extensive experiments on 11 public benchmarks demonstrate that our FS-VFM consistently generalizes better than diverse VFMs, spanning natural and facial domains, fully, weakly, and self-supervised paradigms, small, base, and large ViT scales, and even outperforms SOTA task-specific methods, while FS-Adapter offers an excellent efficiency-performance trade-off. The code and models are available on https://fsfm-3c.github.io/fsvfm.html.

Figures (9)

• Figure 1: A transferable, generalizable, and scalable Face Security Vision Foundation Model (FS-VFM). Simple fine-tuning of the vanilla ViT pre-trained from FS-VFM sets a new generalization bar across various downstream face security tasks, while the FS-Adapter enables ultra-efficient tuning. Results in the line sub-chart are average metrics from \ref{['tab:dfd_vfm']}, \ref{['tab:fas_vfm']}, and \ref{['tab:DiFF']}.
• Figure 2: Overview of FS-VFM self-supervised pre-training framework for learning foundational representations of real faces (3C). Guided by the CRFR-P masking strategy, the masked image modeling (MIM) network promotes intra-region Consistency with $\mathcal{L}_\mathit{rec}^\mathit{m}$ and enforces inter-region Coherency via $\mathcal{L}_\mathit{rec}^\mathit{fr}$, while the instance discrimination (ID) network collaborates to foster local-to-global Correspondence through $\mathcal{L}_\mathit{sim}$. Given an input image $I$, the CRFR-P masking generates a facial region mask $M_\mathit{fr}$ and an image mask $M$ sequentially. The MIM network, a masked autoencoder, reconstructs the masked face $I_\mathit{m}$ from visible patches $x_\mathit{v}$ (masked by $M$), emphasizing the fully masked region $I_\mathit{m}^\mathit{fr}$ (specified by $M_\mathit{fr}$). The ID network maximizes the representation similarity between the masked online view $I_\mathit{v}$ and the full (unmasked) target view $I$ of the same sample by projection onto a disentangled space structured via Siamese representation decoders. After pre-training, the online encoder $E_\mathit{o}$, a vanilla ViT , is applied to boost downstream face security tasks.
• Figure 3: Comparison of masking strategies for face images (75% masking ratio). (a) Simple random masking. (b) Fasking-I, adapted from MARLIN cai2023marlin, priority masking regions $\notin$ {bg, skin}. (c) Our FRP masking for intra-region consistency: Proportional masking within each Facial Region $\in$ {$\mathit{FR}$}. (d) Our CRFR-R masking for inter-region coherency: Covering a Random Facial Region $\in$ {$\mathit{fr}$} and then Random masking other patches. (e) Our CRFR-P masking for both intra-region consistency and inter-region coherency: Covering a Random Facial Region $\in$$\mathit{fr}$ and then Proportional masking other regions $\in$ {$\mathit{FR}-\mathit{fr}$}. All masks are binary (black solely highlights $\mathit{fr}$).
• Figure 4: Mean attention distance dosovitskiy2020image (Top, global ↑) and Kullback-Leibler divergence xie2023revealing (Bottom, diverse ↑) of each attention head (small dot) across all blocks (x-axis) in the MAE he2022masked ViT-B/16 encoder pre-trained by different facial masking strategies, with the average one (large dot) for each block.
• Figure 5: Visualization of the self-attention map averaged across all heads from the last block of the ViT-B/16 encoder pre-trained by MAE he2022masked with different facial masking strategies.