CAD: A General Multimodal Framework for Video Deepfake Detection via Cross-Modal Alignment and Distillation

TL;DR

The paper tackles multimodal video deepfake detection by addressing both modality-specific artifacts and cross-modal semantic misalignments. It introduces CAD, a dual-path framework that combines cross-modal alignment (via frozen CLIP video and Whisper audio with cross-attention and KL divergence) and cross-modal distillation (using LoRA-tuned audio encoders and SimSiam-based losses) to maximize mutual information across modalities while preserving modality-specific cues. The approach formalizes the learning objective around $I(x_1,x_2,y)$ and $H(x_1,y|x_2)$, enabling joint and decoupled representations, and demonstrates state-of-the-art results on IDForge and FakeAVCeleb with extensive ablations and visual analyses. The findings suggest that harmonious integration of modality-specific traces with cross-modal coherence yields robust detection against evolving multimodal forgeries, with practical impact for safer media authentication.

Abstract

The rapid emergence of multimodal deepfakes (visual and auditory content are manipulated in concert) undermines the reliability of existing detectors that rely solely on modality-specific artifacts or cross-modal inconsistencies. In this work, we first demonstrate that modality-specific forensic traces (e.g., face-swap artifacts or spectral distortions) and modality-shared semantic misalignments (e.g., lip-speech asynchrony) offer complementary evidence, and that neglecting either aspect limits detection performance. Existing approaches either naively fuse modality-specific features without reconciling their conflicting characteristics or focus predominantly on semantic misalignment at the expense of modality-specific fine-grained artifact cues. To address these shortcomings, we propose a general multimodal framework for video deepfake detection via Cross-Modal Alignment and Distillation (CAD). CAD comprises two core components: 1) Cross-modal alignment that identifies inconsistencies in high-level semantic synchronization (e.g., lip-speech mismatches); 2) Cross-modal distillation that mitigates feature conflicts during fusion while preserving modality-specific forensic traces (e.g., spectral distortions in synthetic audio). Extensive experiments on both multimodal and unimodal (e.g., image-only/video-only)deepfake benchmarks demonstrate that CAD significantly outperforms previous methods, validating the necessity of harmonious integration of multimodal complementary information.

Figures (3)

• Figure 1: Venn diagram illustrating different types of artifacts in multimodal deepfakes, categorized into modality-specific and modality-shared cues. Specifically, visual artifacts ($x_1$) may include blending boundaries of face-swapping, while audio artifacts ($x_2$) might exhibit spectral anomalies. The shared space ($x_{12}$) captures semantic cross-modal mismatches, such as inconsistency between lip movements and speech. Ideally, a robust system should integrate both perspectives for improved accuracy.
• Figure 2: The overview of CAD. Our proposed CAD is designed to maximize and fully mine both modality-specific and modality-shared cues for robust deepfake detection.
• Figure 3: Visual illustrations of our method. Left: The visualization results of modality-specific learning and modality-shared learning by CAM Zhou2015LearningDF.Origin denotes the original video input. Modality-specific shows the attention distribution within the vision unimodal encoder, where attention focuses mainly on visual artifacts. Instead, Modality-shared illustrates the attention distribution when both modalities are aligned, with attention primarily on the lips and surrounding musculature. Right: t-SNE visualization on integrated embeddings.

Theorems & Definitions (5)

• Lemma 1
• Lemma 2
• Lemma 3
• Theorem 1
• proof