EDVD-LLaMA: Explainable Deepfake Video Detection via Multimodal Large Language Model Reasoning

TL;DR

This work tackles the need for explainable deepfake video detection by introducing EDVD-LLaMA, a multimodal reasoning framework that produces traceable explanations alongside detection results. It combines Spatio-Temporal Subtle Information Tokenization (ST-SIT) with Fine-grained Multimodal Chain-of-Thought (Fg-MCoT) to leverage both local deepfake cues and global semantic context, guided by structured facial metrics. A new dataset, ER-FF++set, provides dual supervision for detection and reasoning, enabling robust generalization across forgery methods and datasets. Extensive experiments show superior detection accuracy and explainability compared with existing MLLMs, highlighting the practical value of transparent, region-aware deepfake analysis. Overall, EDVD-LLaMA advances explainable multimedia forensics by delivering reliable, interpretable reasoning in deepfake video detection.

Abstract

The rapid development of deepfake video technology has not only facilitated artistic creation but also made it easier to spread misinformation. Traditional deepfake video detection (DVD) methods face issues such as a lack of transparency in their principles and insufficient generalization capabilities to cope with evolving forgery techniques. This highlights an urgent need for detectors that can identify forged content and provide verifiable reasoning explanations. This paper proposes the explainable deepfake video detection (EDVD) task and designs the EDVD-LLaMA multimodal, a large language model (MLLM) reasoning framework, which provides traceable reasoning processes alongside accurate detection results and trustworthy explanations. Our approach first incorporates a Spatio-Temporal Subtle Information Tokenization (ST-SIT) to extract and fuse global and local cross-frame deepfake features, providing rich spatio-temporal semantic information input for MLLM reasoning. Second, we construct a Fine-grained Multimodal Chain-of-Thought (Fg-MCoT) mechanism, which introduces facial feature data as hard constraints during the reasoning process to achieve pixel-level spatio-temporal video localization, suppress hallucinated outputs, and enhance the reliability of the chain of thought. In addition, we build an Explainable Reasoning FF++ dataset (ER-FF++set), leveraging structured data to annotate videos and ensure quality control, thereby supporting dual supervision for reasoning and detection. Extensive experiments demonstrate that EDVD-LLaMA achieves outstanding performance and robustness in terms of detection accuracy, explainability, and its ability to handle cross-forgery methods and cross-dataset scenarios. Compared to previous DVD methods, it provides a more explainable and superior solution. The project page is available at: https://11ouo1.github.io/edvd-llama/.

Figures (5)

• Figure 1: Left: Performance comparison between EDVD-LLaMA and MLLMs on the DVD task. Right: Performance comparison between EDVD-LLaMA and traditional methods on cross-forgery and cross-dataset detection tasks. EDVD-LLaMA demonstrates superior performance in the above tasks.
• Figure 2: (a) Conventional deepfake video detection: only outputs a binary real/fake label. (b) Our method (MLLM + Fg-MCoT) fuses consecutive frame images and fine-grained facial landmarks to generate a multimodal reasoning chain, enabling a multimodal large language model to achieve multimodal interaction and provide explicit continuous reasoning evidence. "(omitted)" indicates content omitted for brevity.
• Figure 3: EDVD-LLaMA pipeline. The candidate video is processed according to a sampling strategy to obtain video clip $\mathcal{V}_{\text{clip}}$, which is passed through the spatio-temporal subtle information tokenization $\mathcal{E}_{\text{st}}$ to produce the fused visual representation $\mathcal{T}_{\text{vid}}$. In parallel, a face landmark detection module $\mathcal{E}_{\text{fl}}$ extract frame-wise keypoints and related statistics $\mathcal{T}_{\text{fac}}=\{\mathcal{M}_{\text{c}}, \mathcal{M}_{\Delta}\}$. Then, $\mathcal{T}_{\text{vid}}$ and $\mathcal{T}_{\text{fac}}$ are combined with a thought prompt $\mathcal{P}_{\text{tht}}$ as input to the LLM in Stage 1 to generate an explanatory rationale $\mathcal{R}_{\text{c}}$. The main LLM in Stage 2 receives $\mathcal{T}_{\text{vid}}$, $\mathcal{R}_{\text{c}}$, and the user instruction $\mathcal{P}_{\text{qt}}$, and outputs a structured <think> reasoning trace and an <answer> decision label, achieving explainable multimodal deepfake video detection and reasoning. "omitted" indicates content omitted for brevity.
• Figure 4: Construction pipeline of the proposed ER-FF++set. We sample videos for multiple manipulation types (Deepfakes, Face2Face, FaceShifter, FaceSwap) from FaceForensics++, and then apply manipulation-type–specific prompts to steer the Assistant LLM in producing structured rationales for each decision.
• Figure 5: Qualitative results on ER-FF++set. We compare EDVD-LLaMA's test results on deepfake and real videos from ER-FF++ with VideoLLaVA, VideoChatGPT, VideoChat, and VideoLLaMA3. Green boxes indicate correct judgments, and red boxes indicate incorrect decisions. "omitted" indicates content omitted for brevity. More qualitative results of our model can be found in the supplementary video.