MLLM-Enhanced Face Forgery Detection: A Vision-Language Fusion Solution

TL;DR

This work tackles face forgery detection by leveraging Multimodal Large Language Models (MLLMs) and addresses the integration gap between visual and textual cues. It introduces EFF++ to provide frame-level, explainable annotations via CFAD and MTS, guiding LLM-driven insights. The core architecture, VLF-Net, combines an external detector, an MLLM, and a Vision-Language Fusion Module with cross-attention to fuse visual and textual representations in a three-stage training pipeline. On FF++ and cross-dataset benchmarks (CDF2, DFDC, DFDCP, FFIW), VLF-FFD achieves state-of-the-art performance and delivers interpretable explanations that align with attended regions, demonstrating strong generalization and practical impact for detecting manipulated faces.

Abstract

Reliable face forgery detection algorithms are crucial for countering the growing threat of deepfake-driven disinformation. Previous research has demonstrated the potential of Multimodal Large Language Models (MLLMs) in identifying manipulated faces. However, existing methods typically depend on either the Large Language Model (LLM) alone or an external detector to generate classification results, which often leads to sub-optimal integration of visual and textual modalities. In this paper, we propose VLF-FFD, a novel Vision-Language Fusion solution for MLLM-enhanced Face Forgery Detection. Our key contributions are twofold. First, we present EFF++, a frame-level, explainability-driven extension of the widely used FaceForensics++ (FF++) dataset. In EFF++, each manipulated video frame is paired with a textual annotation that describes both the forgery artifacts and the specific manipulation technique applied, enabling more effective and informative MLLM training. Second, we design a Vision-Language Fusion Network (VLF-Net) that promotes bidirectional interaction between visual and textual features, supported by a three-stage training pipeline to fully leverage its potential. VLF-FFD achieves state-of-the-art (SOTA) performance in both cross-dataset and intra-dataset evaluations, underscoring its exceptional effectiveness in face forgery detection.

Figures (8)

• Figure 1: Comparative overview of face forgery detection frameworks evaluated on the Celeb-DeepFake-v2 (CDF2) benchmark Li_2020_CVPR. (a) Traditional methods use face forgery detectors exclusively for classification, achieving a cross-dataset AUC of 95.50%. (b)$\mathcal{X}^2$-DFD chen2024textit, representing the first category of MLLM-based approaches, leverages an LLM (also MLLM) for both classification and explanation, also reaching 95.50% AUC. (c) M2F2-Det guo2025rethinking, belonging to the second category of MLLM-based methods, utilizes an external detector for classification and an LLM for explanation, achieving an AUC of 95.10%. (d) The proposed VLF-FFD enables bidirectional interaction between visual and textual modalities, delivering superior performance with a 97.29% AUC on CDF2.
• Figure 2: Annotation for a fake video frame from the EFF++ dataset. CFAD is employed to identify and explain facial forgeries, while MTS is used to extract and summarize the specific manipulation strategy applied. The combination of these methods results in an informative textual annotation.
• Figure 3: Overview of the VLF-Net architecture. The network consists of three main components: (1) an external detector, (2) an MLLM that includes an image encoder, two projectors, and an LLM, and (3) the innovative Vision-Language Fusion Module (VLFM). VLF-Net facilitates bidirectional interaction between visual and textual information. GAP1D refers to Global Average Pooling 1D.
• Figure 4: Overview of the three-stage training pipeline for VLF-Net. In Stage 1, the external detector is trained independently. In Stage 2, the external detector is frozen while the MLLM is fine-tuned. In Stage 3, both the external detector and the MLLM are frozen, and the VLFM is trained.
• Figure 5: Attention maps generated by VLFM alongside their corresponding explanatory outputs for examples from the FF++ dataset. The attention maps highlight facial regions containing forgery artifacts, closely matching the areas mentioned in the explanatory contents. Notably, this visualization demonstrates the strong alignment between visual and textual modalities achieved by VLF-FFD.