Guard Me If You Know Me: Protecting Specific Face-Identity from Deepfakes

TL;DR

VIPGuard addresses the privacy and security risks of deepfakes targeting known individuals by modeling targeted, identity-aware detection as a fine-grained face recognition task. The framework jointly learns global identity priors and fine-grained facial attributes via a three-stage MLLM-based pipeline and introduces a lightweight VIP token for per-user customization, enabling explainable predictions. To support evaluation, the paper introduces VIPBench, a comprehensive identity-centric benchmark spanning 22 VIPs and 80k images across 14 manipulation methods. Experiments show VIPGuard outperforms general detectors and other LLM-based approaches, with strong performance even in annotation-free scenarios and under degradations, signaling practical applicability for protecting high-risk identities. The work advances personalized deepfake defense, offering robust, explainable, and scalable solutions for safeguarding public figures and other VIPs against identity-based forgery threats.

Abstract

Securing personal identity against deepfake attacks is increasingly critical in the digital age, especially for celebrities and political figures whose faces are easily accessible and frequently targeted. Most existing deepfake detection methods focus on general-purpose scenarios and often ignore the valuable prior knowledge of known facial identities, e.g., "VIP individuals" whose authentic facial data are already available. In this paper, we propose \textbf{VIPGuard}, a unified multimodal framework designed to capture fine-grained and comprehensive facial representations of a given identity, compare them against potentially fake or similar-looking faces, and reason over these comparisons to make accurate and explainable predictions. Specifically, our framework consists of three main stages. First, fine-tune a multimodal large language model (MLLM) to learn detailed and structural facial attributes. Second, we perform identity-level discriminative learning to enable the model to distinguish subtle differences between highly similar faces, including real and fake variations. Finally, we introduce user-specific customization, where we model the unique characteristics of the target face identity and perform semantic reasoning via MLLM to enable personalized and explainable deepfake detection. Our framework shows clear advantages over previous detection works, where traditional detectors mainly rely on low-level visual cues and provide no human-understandable explanations, while other MLLM-based models often lack a detailed understanding of specific face identities. To facilitate the evaluation of our method, we built a comprehensive identity-aware benchmark called \textbf{VIPBench} for personalized deepfake detection, involving the latest 7 face-swapping and 7 entire face synthesis techniques for generation. The code is available at https://github.com/KQL11/VIPGuard .

Figures (19)

• Figure 1: An illustrative comparison between natural images (from LAION-Face laion_face) and images generated by GPT-4o gpt4o, showing localized inconsistencies in facial attributes, such as eye pouches and facial shapes.
• Figure 2: Overview of the data collection, VIP-Guard, and Evaluation. VIPGuard's training and inference pipeline for facial attribute understanding, identity discrimination, and VIP user customization.
• Figure 3: Illustration of the proposed VIPBench, which includes three personalized datasets, (a) Facial Attributes Description Dataset $\mathcal{D}_{FA}$, (b) Identity Discrimination Dataset $\mathcal{D}_{ID}$, and (c) VIPEval $\mathcal{D}_{Eval}$. (d) Some examples of the facial attributes used in the $\mathcal{D}_{FA}$ are also illustrated here, while the full set is available in the supplementary material. The real images shown in (c) are from CelebDF laion_face and VIPBench, while the fake ones are generated using multiple models. All images in (d) are sourced from LAION-Face laion_face.
• Figure 4: Illustration of the three stages of training the proposed VIPGuard framework.
• Figure 5: Visual illustration of the analysis of VIP-Guard detecting anomalous local facial attributes for EFS (left) and FS (right). The two real images are sourced from LAION-Face laion_face, while the fake images in the left and right subfigures were generated by GPT-4o gpt4o and HifiFace wang2021hififace, respectively.