Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Divide and Conquer: Multimodal Video Deepfake Detection via Cross-Modal Fusion and Localization

Qingcao Li, Miao He, Liang Yi, Qing Wen, Yitao Zhang, Hongshuo Jin, Peng Cheng, Zhongjie Ba, Li Lu, Kui Ren

TL;DR

This work tackles the detection and localization of audio-visual deepfakes in Track 2 of the DDL-Challenge by deploying three modality-specific modules (audio detection, audio localization, and image detection/localization) coupled with two fusion stages for detection and localization. The audio branch leverages Wav2Vec2.0-AASIST and Boundary-aware BAM with a dual-loss training regime, while the image branch uses Xception with targeted augmentation and a frame-level detection/localization approach. Through interval-wise fusion, the system effectively fuses complementary cues across modalities, yielding a final test performance of AUC 0.87, AP 0.55, and AR 0.23 for the fusion-based localization. The results demonstrate that cross-modal fusion significantly improves robustness to partial forgeries and temporal misalignments, highlighting the practical value of modular multimodal detection pipelines for real-world deepfake scenarios.

Abstract

This paper presents a system for detecting fake audio-visual content (i.e., video deepfake), developed for Track 2 of the DDL Challenge. The proposed system employs a two-stage framework, comprising unimodal detection and multimodal score fusion. Specifically, it incorporates an audio deepfake detection module and an audio localization module to analyze and pinpoint manipulated segments in the audio stream. In parallel, an image-based deepfake detection and localization module is employed to process the visual modality. To effectively leverage complementary information across different modalities, we further propose a multimodal score fusion strategy that integrates the outputs from both audio and visual modules. Guided by a detailed analysis of the training and evaluation dataset, we explore and evaluate several score calculation and fusion strategies to improve system robustness. Overall, the final fusion-based system achieves an AUC of 0.87, an AP of 0.55, and an AR of 0.23 on the challenge test set, resulting in a final score of 0.5528.

Divide and Conquer: Multimodal Video Deepfake Detection via Cross-Modal Fusion and Localization

TL;DR

This work tackles the detection and localization of audio-visual deepfakes in Track 2 of the DDL-Challenge by deploying three modality-specific modules (audio detection, audio localization, and image detection/localization) coupled with two fusion stages for detection and localization. The audio branch leverages Wav2Vec2.0-AASIST and Boundary-aware BAM with a dual-loss training regime, while the image branch uses Xception with targeted augmentation and a frame-level detection/localization approach. Through interval-wise fusion, the system effectively fuses complementary cues across modalities, yielding a final test performance of AUC 0.87, AP 0.55, and AR 0.23 for the fusion-based localization. The results demonstrate that cross-modal fusion significantly improves robustness to partial forgeries and temporal misalignments, highlighting the practical value of modular multimodal detection pipelines for real-world deepfake scenarios.

Abstract

This paper presents a system for detecting fake audio-visual content (i.e., video deepfake), developed for Track 2 of the DDL Challenge. The proposed system employs a two-stage framework, comprising unimodal detection and multimodal score fusion. Specifically, it incorporates an audio deepfake detection module and an audio localization module to analyze and pinpoint manipulated segments in the audio stream. In parallel, an image-based deepfake detection and localization module is employed to process the visual modality. To effectively leverage complementary information across different modalities, we further propose a multimodal score fusion strategy that integrates the outputs from both audio and visual modules. Guided by a detailed analysis of the training and evaluation dataset, we explore and evaluate several score calculation and fusion strategies to improve system robustness. Overall, the final fusion-based system achieves an AUC of 0.87, an AP of 0.55, and an AR of 0.23 on the challenge test set, resulting in a final score of 0.5528.
Paper Structure (16 sections, 4 equations, 2 figures, 4 tables, 2 algorithms)

This paper contains 16 sections, 4 equations, 2 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Data Analysis
  3. Method
  4. Audio
  5. Detection
  6. Localization
  7. Image
  8. Training Strategy
  9. Inference Strategy
  10. Fusion
  11. Audio-Visual Multimodal Fusion for Detection
  12. Audio-Visual Multimodal Fusion for Localization
  13. Evaluation
  14. Audio-Visual Multimodal Fusion for Detection
  15. Audio-Visual Multimodal Fusion for Localization
  16. ...and 1 more sections

Figures (2)

  • Figure 1: Overview of the proposed audio-visual deepfake detection and localization framework. The system is composed of three modality-specific modules and two score fusion units. The Audio Detection Module outputs a single video-level confidence score indicating the presence of audio tampering. The Audio Localization Module produces frame-level forgery likelihoods, which are aggregated to generate interval-level scores based on pre-defined temporal segments. The Image Detection and Localization Module concurrently provides both video-level detection and frame-wise visual tampering scores. For detection, video-level scores from the audio and image branches are fused via the Detection Score Fusion. For localization, frame- or interval-level outputs from both audio and image streams are aligned and merged by the Localization Score Fusion to generate a unified temporal localization result.
  • Figure 2: The overall architecture of BAM.