Beyond Face Swapping: A Diffusion-Based Digital Human Benchmark for Multimodal Deepfake Detection

TL;DR

Diffusion-based digital humans enable photorealistic, temporally coherent forgeries that outpace traditional detectors trained on older benchmarks. The authors introduce DigiFakeAV, a large-scale, multimodal benchmark with 60,000 videos (8.4 million frames) generated by five diffusion models and a voice clone, featuring diverse demographics and real-world scenarios; misrecognition by humans can reach 68% and existing detectors suffer major AUC drops. To counter this, they propose DigiShield, a spatiotemporal, cross-modal fusion detector that jointly analyzes video and audio through two-stream architectures and attention-based fusion while optimizing with contrastive and cross-entropy losses. DigiShield achieves 80.1% AUC on DigiFakeAV and generalizes to DF-TIMIT with 100% AUC, illustrating the value of multimodal temporal modeling for next-generation deepfake detection and setting a new baseline for future work.

Abstract

In recent years, the explosive advancement of deepfake technology has posed a critical and escalating threat to public security: diffusion-based digital human generation. Unlike traditional face manipulation methods, such models can generate highly realistic videos with consistency via multimodal control signals. Their flexibility and covertness pose severe challenges to existing detection strategies. To bridge this gap, we introduce DigiFakeAV, the new large-scale multimodal digital human forgery dataset based on diffusion models. Leveraging five of the latest digital human generation methods and a voice cloning method, we systematically construct a dataset comprising 60,000 videos (8.4 million frames), covering multiple nationalities, skin tones, genders, and real-world scenarios, significantly enhancing data diversity and realism. User studies demonstrate that the misrecognition rate by participants for DigiFakeAV reaches as high as 68%. Moreover, the substantial performance degradation of existing detection models on our dataset further highlights its challenges. To address this problem, we propose DigiShield, an effective detection baseline based on spatiotemporal and cross-modal fusion. By jointly modeling the 3D spatiotemporal features of videos and the semantic-acoustic features of audio, DigiShield achieves state-of-the-art (SOTA) performance on the DigiFakeAV and shows strong generalization on other datasets.

Figures (2)

• Figure 1: Workflow diagram for the construction of DigiFakeAV.
• Figure 2: Overview of the DigiShield, it detects forgery traces by integrating spatiotemporal information from both video and audio modalities.