MARE: Multimodal Alignment and Reinforcement for Explainable Deepfake Detection via Vision-Language Models

TL;DR

MARE addresses the challenge of explainable Deepfake detection by integrating vision-language reasoning with reinforcement learning from human feedback and a forgery-focused representation disentanglement. It introduces a Deepfake multimodal alignment dataset (DMA) and a Forgery Disentanglement Module (FDM) to separate intrinsic forgery traces from high-level semantics, while a multi-component reward system guides the VLM to produce text that is spatially aligned with evidence. Empirical results show state-of-the-art detection performance on challenging datasets (notably WDF and DFDC) and superior reasoning accuracy (Acc and F1) on the DMA, with qualitative evidence of tighter text-box alignment and reduced hallucinations. The work advances practical, explainable Deepfake forensic tools by delivering region-aware explanations that align with human preferences and evidence grounding.

Abstract

Deepfake detection is a widely researched topic that is crucial for combating the spread of malicious content, with existing methods mainly modeling the problem as classification or spatial localization. The rapid advancements in generative models impose new demands on Deepfake detection. In this paper, we propose multimodal alignment and reinforcement for explainable Deepfake detection via vision-language models, termed MARE, which aims to enhance the accuracy and reliability of Vision-Language Models (VLMs) in Deepfake detection and reasoning. Specifically, MARE designs comprehensive reward functions, incorporating reinforcement learning from human feedback (RLHF), to incentivize the generation of text-spatially aligned reasoning content that adheres to human preferences. Besides, MARE introduces a forgery disentanglement module to capture intrinsic forgery traces from high-level facial semantics, thereby improving its authenticity detection capability. We conduct thorough evaluations on the reasoning content generated by MARE. Both quantitative and qualitative experimental results demonstrate that MARE achieves state-of-the-art performance in terms of accuracy and reliability.

Figures (8)

• Figure 1: Illustration of different Deepfake detection methods. The conventional Deepfake detectors provide a discriminative decision. The existing LLM-based methods generate plain textual reasoning content. While the pre-trained VLMs struggle to satisfy Deepfake reasoning demands. Our method generates detailed forgery traces analysis and provides spatial localization information for supporting evidence.
• Figure 2: Diagrammatic overview of the proposed MARE framework. The reward functions and RLHF algorithm are displayed on the right side (see Section \ref{['sec:reward_func']} for details). The vision encoder, text tokenizer, and large language model decoder are key components of VLM. The forgery disentanglement module is introduced for forgery traces extraction(see Section \ref{['sec_FDM']} for details). During the inference phase, MARE generates text-spatially aligned reasoning content for a given image-text query.
• Figure 3: Examples of the Deepfake multimodal alignment dataset (DMA).
• Figure 4: Illustration of the proposed forgery disentanglement detector (FDM). The diagrammatic structure of FDM is shown in the left panel, while the right panel depicts the structure of the adversarial classifier.
• Figure 5: Examples of reasoning content generated by Qwen, GPT-5, M2F2, and MARE.