DeepfakeBench-MM: A Comprehensive Benchmark for Multimodal Deepfake Detection

TL;DR

This work tackles the rising threat of multimodal deepfakes by introducing Mega-MMDF, a million-scale, diverse audiovisual forgery dataset built from 21 forgery pipelines and 28 methods, together with DeepfakeBench-MM, a unified, extensible benchmark that standardizes preprocessing, training, and evaluation for MM-DFD. The authors implement 11 detectors across baselines, regular models, ensembles, and pretrained multimodal LLMs, and evaluate them on five datasets under intra-, cross-dataset, and cross-pipeline settings, using AUC as the primary metric. Key findings include strong in-domain gains with finetuning, notable cross-dataset generalization gaps especially on partial forgeries, and a consistent visual-bias across modalities that motivates bias-mitigation strategies such as modality masking. The proposed infrastructure enables fair, scalable comparisons and provides actionable insights—such as artifact dominance (EFS>FS>FR) and the benefits of targeted backbone fine-tuning—that can guide future MM-DFD research and dataset design, with broad implications for real-world detection systems.

Abstract

The misuse of advanced generative AI models has resulted in the widespread proliferation of falsified data, particularly forged human-centric audiovisual content, which poses substantial societal risks (e.g., financial fraud and social instability). In response to this growing threat, several works have preliminarily explored countermeasures. However, the lack of sufficient and diverse training data, along with the absence of a standardized benchmark, hinder deeper exploration. To address this challenge, we first build Mega-MMDF, a large-scale, diverse, and high-quality dataset for multimodal deepfake detection. Specifically, we employ 21 forgery pipelines through the combination of 10 audio forgery methods, 12 visual forgery methods, and 6 audio-driven face reenactment methods. Mega-MMDF currently contains 0.1 million real samples and 1.1 million forged samples, making it one of the largest and most diverse multimodal deepfake datasets, with plans for continuous expansion. Building on it, we present DeepfakeBench-MM, the first unified benchmark for multimodal deepfake detection. It establishes standardized protocols across the entire detection pipeline and serves as a versatile platform for evaluating existing methods as well as exploring novel approaches. DeepfakeBench-MM currently supports 5 datasets and 11 multimodal deepfake detectors. Furthermore, our comprehensive evaluations and in-depth analyses uncover several key findings from multiple perspectives (e.g., augmentation, stacked forgery). We believe that DeepfakeBench-MM, together with our large-scale Mega-MMDF, will serve as foundational infrastructures for advancing multimodal deepfake detection.

Figures (7)

• Figure 1: Left: Statistics of our Mega-MMDF dataset, demonstrating significant advantages in scale and forgery diversity. Note that only audiovisual samples in each dataset are counted. Right: Distributions of gender (top) and skin tone (bottom) in train/validation/test sets of Mega-MMDF.
• Figure 1: Overall quality assessments of different datasets. The best results are highlighted in bold.
• Figure 2: Illustration of the multimodal forgery pipeline, showing the steps of data construction.
• Figure 3: Cross-pipeline One-Versus-All (OvA) protocol. Left: The cross-pipeline performance heatmap, where red bounding boxes highlight generalization of pipelines employing the same visual forgeries. Right: Detailed illustrations of 22 pipelines in our Mega-MMDF dataset.
• Figure 4: t-SNE visualizations of the baseline model on the cross-pipeline setting.