Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multiscale Adaptive Conflict-Balancing Model For Multimedia Deepfake Detection

Zihan Xiong, Xiaohua Wu, Lei Chen, Fangqi Lou

TL;DR

MACB-DF tackles audiovisual deepfake detection by addressing modality bias and gradient conflicts in multimodal fusion. It combines a multi-scale feature extractor with adaptive contrastive learning and a Pareto-based gradient balancing module to preserve unimodal information while harmonizing cross-modal representations. The approach achieves state-of-the-art results across DefakeAVMiT, FakeAVCeleb, and DFDC, with average ACC around 95.5% and robust cross-dataset generalization, including notable gains on cross-domain tests. These advances enhance detection reliability and practical applicability in security, authentication, and media review systems by delivering balanced, high-fidelity multimodal representations.

Abstract

Advances in computer vision and deep learning have blurred the line between deepfakes and authentic media, undermining multimedia credibility through audio-visual forgery. Current multimodal detection methods remain limited by unbalanced learning between modalities. To tackle this issue, we propose an Audio-Visual Joint Learning Method (MACB-DF) to better mitigate modality conflicts and neglect by leveraging contrastive learning to assist in multi-level and cross-modal fusion, thereby fully balancing and exploiting information from each modality. Additionally, we designed an orthogonalization-multimodal pareto module that preserves unimodal information while addressing gradient conflicts in audio-video encoders caused by differing optimization targets of the loss functions. Extensive experiments and ablation studies conducted on mainstream deepfake datasets demonstrate consistent performance gains of our model across key evaluation metrics, achieving an average accuracy of 95.5% across multiple datasets. Notably, our method exhibits superior cross-dataset generalization capabilities, with absolute improvements of 8.0% and 7.7% in ACC scores over the previous best-performing approach when trained on DFDC and tested on DefakeAVMiT and FakeAVCeleb datasets.

Multiscale Adaptive Conflict-Balancing Model For Multimedia Deepfake Detection

TL;DR

MACB-DF tackles audiovisual deepfake detection by addressing modality bias and gradient conflicts in multimodal fusion. It combines a multi-scale feature extractor with adaptive contrastive learning and a Pareto-based gradient balancing module to preserve unimodal information while harmonizing cross-modal representations. The approach achieves state-of-the-art results across DefakeAVMiT, FakeAVCeleb, and DFDC, with average ACC around 95.5% and robust cross-dataset generalization, including notable gains on cross-domain tests. These advances enhance detection reliability and practical applicability in security, authentication, and media review systems by delivering balanced, high-fidelity multimodal representations.

Abstract

Advances in computer vision and deep learning have blurred the line between deepfakes and authentic media, undermining multimedia credibility through audio-visual forgery. Current multimodal detection methods remain limited by unbalanced learning between modalities. To tackle this issue, we propose an Audio-Visual Joint Learning Method (MACB-DF) to better mitigate modality conflicts and neglect by leveraging contrastive learning to assist in multi-level and cross-modal fusion, thereby fully balancing and exploiting information from each modality. Additionally, we designed an orthogonalization-multimodal pareto module that preserves unimodal information while addressing gradient conflicts in audio-video encoders caused by differing optimization targets of the loss functions. Extensive experiments and ablation studies conducted on mainstream deepfake datasets demonstrate consistent performance gains of our model across key evaluation metrics, achieving an average accuracy of 95.5% across multiple datasets. Notably, our method exhibits superior cross-dataset generalization capabilities, with absolute improvements of 8.0% and 7.7% in ACC scores over the previous best-performing approach when trained on DFDC and tested on DefakeAVMiT and FakeAVCeleb datasets.
Paper Structure (25 sections, 20 equations, 7 figures, 3 tables)

This paper contains 25 sections, 20 equations, 7 figures, 3 tables.

Table of Contents

  1. INTRODDUCTION
  2. RELATED WORK
  3. Unimodal Deepfake Detection Methods
  4. Unimodal Bias in Multimodal Deep Learning
  5. METHOD
  6. Feature Extract
  7. Multi-Scale Feature Fusion With Contrastive Learning
  8. Multi-modal Adaptive Contrast Learning
  9. Adaptive Parameter Optimization for Contrastive Learning
  10. Multi-Scale Feature Extraction
  11. Multimodal Feature Fusion
  12. Representation Fusion via Multi-Head Attention Framework
  13. Classify
  14. Orthogonalization-multimodal Pareto
  15. EXPERIMENTS
  16. ...and 10 more sections

Figures (7)

  • Figure 1: Multiscale Adaptive Conflict-Balancing model for multimedia Deepfake detection deptly balances conflicts and biases among modalities, thereby fully unlocking the potential of intra-modality and inter-modality information. There are five stages in this method: Feature Extract, Multi-Scale Feature Fusion With Contrastive Learning, RFMF, OM-Pareto, Classify
  • Figure 2: Adaptive Contrast Learning
  • Figure 3: Orthogonalization-multimodal Pareto
  • Figure 4: Ablation Study on Contrastive Learning: Latent Space Visualization
  • Figure 5: Ablation Study on MFFCL: Impact on AUC/ACC Metrics for DFDC and FakeAVCeleb Datasets
  • ...and 2 more figures