Rethinking Vision-Language Model in Face Forensics: Multi-Modal Interpretable Forged Face Detector

TL;DR

The paper addresses the challenge of detecting deepfakes while simultaneously providing interpretable explanations. It introduces M2F2-Det, a multi-modal detector that leverages a frozen CLIP image encoder, a deepfake encoder, and an LLM via a Bridge Adapter, enhanced by Forgery Prompt Learning to adapt CLIP for forgery detection. The approach yields state-of-the-art results on six detection datasets and achieves top performance in explanation generation on the DD-VQA benchmark, while offering faithful textual rationales grounded in forged attention maps. This combination of robust open-set visual understanding and natural-language explanations improves trust and generalization, enabling more reliable deployment in real-world media forensics.

Abstract

Deepfake detection is a long-established research topic vital for mitigating the spread of malicious misinformation. Unlike prior methods that provide either binary classification results or textual explanations separately, we introduce a novel method capable of generating both simultaneously. Our method harnesses the multi-modal learning capability of the pre-trained CLIP and the unprecedented interpretability of large language models (LLMs) to enhance both the generalization and explainability of deepfake detection. Specifically, we introduce a multi-modal face forgery detector (M2F2-Det) that employs tailored face forgery prompt learning, incorporating the pre-trained CLIP to improve generalization to unseen forgeries. Also, M2F2-Det incorporates an LLM to provide detailed textual explanations of its detection decisions, enhancing interpretability by bridging the gap between natural language and subtle cues of facial forgeries. Empirically, we evaluate M2F2-Det on both detection and explanation generation tasks, where it achieves state-of-the-art performance, demonstrating its effectiveness in identifying and explaining diverse forgeries.

Figures (7)

• Figure 1: (a) and (b) represent conventional deepfake detectors and DDVQA-BLIP zhang2024common, which take an image as the input and output the fake probability (e.g., score) and textual explanations, respectively. (c) In this work, we propose a multi-modal face forgery detector (M2F2-Det) that produces both fake probability and textual explanations.
• Figure 2: (a) Forgery Prompt Learning (FPL) adapts CLIP to deepfake detection by optimizing UF-prompts and layer-wise forgery tokens (LF tokens). UF-prompts consist of three segments: trainable general forgery tokens (i.e., $\mathbf{V}_1$, $\mathbf{V}_2$, and $\mathbf{V}_3$), specific forgery tokens (i.e., $\mathbf{V}_4$, $\mathbf{V}_5$, and $\mathbf{V}_6$), and a fixed textual description "Forged Face". LF tokens are introduced in the CLIP text encoder and detailed in Fig. \ref{['fig_archi']}b and Sec. \ref{['subsec:forgery_prompt_learn']}. (b) The Bridge Adapter connects the CLIP image encoder to the deepfake detector. It integrates with an LLM and a classification head, which output textual explanations and a predicted fake score, respectively.
• Figure 3: (a) The multi-modal face forgery detector (M2F2-Det) comprises pre-trained CLIP image and text encoders (i.e., $\mathcal{E}_{I}$ and $\mathcal{E}_{T}$), a deepfake encoder, as well as an LLM. Given the universal forgery prompts (UF-prompts) as input, $\mathcal{E}_{T}$ generates a global text embedding, e.g., $\mathbf{g}^\text{T}$, that guides the generation of a forgery attention mask, e.g., $\mathbf{M}_b$. The deepfake encoder utilizes the bridge adapter, i.e., $\mathcal{E}_A$, for detecting face forgeries (Sec. \ref{['subsec:forgery_loc_det']}), while the LLM generates explanations conditioned on a frequency token $\mathbf{H}_\text{F}$ transformed from the forgery representation ($\mathbf{F}^0$) (Sec. \ref{['subsec:ff_it']}). (b) In the CLIP text encoder, we introduce trainable layer-wise forgery tokens as inputs to each Transformer vaswani2017attention encoder layer.
• Figure 4: The illustration on the Bridge Adapter, in which $\Theta$ represents the transformation conducted by Eq. \ref{['eq:conv_pool']}. [Key: T-Block: transformer encoder block; Block: convolution block.]
• Figure 5: Explanation generation performance on DD-VQA. (a) Judgment performance. [Key: Best results, Acc: Accuracy, F1: F1 Score] (b) Explanation quality is measured by $5$ metrics. (c) Visualizations of forged attentions.