Revealing the Truth with ConLLM for Detecting Multi-Modal Deepfakes

TL;DR

ConLLM tackles multimodal deepfake detection by addressing modality fragmentation and shallow inter-modal reasoning through a two-stage approach that separates feature extraction from semantic integration. It first collects modality-specific embeddings via pre-trained models, then aligns these representations with contrastive learning and refines them with an LLM-based module to reason over cross-modal inconsistencies. The method achieves state-of-the-art results across audio, video, and audio-visual tasks, with substantial gains in EER, ACC, and AUC, and demonstrates favorable computational efficiency relative to large multimodal models. The work highlights practical impact for media integrity, supporting robust, scalable detection with cross-lingual generalization, while noting limitations related to data quality and deployment constraints.

Abstract

The rapid rise of deepfake technology poses a severe threat to social and political stability by enabling hyper-realistic synthetic media capable of manipulating public perception. However, existing detection methods struggle with two core limitations: (1) modality fragmentation, which leads to poor generalization across diverse and adversarial deepfake modalities; and (2) shallow inter-modal reasoning, resulting in limited detection of fine-grained semantic inconsistencies. To address these, we propose ConLLM (Contrastive Learning with Large Language Models), a hybrid framework for robust multimodal deepfake detection. ConLLM employs a two-stage architecture: stage 1 uses Pre-Trained Models (PTMs) to extract modality-specific embeddings; stage 2 aligns these embeddings via contrastive learning to mitigate modality fragmentation, and refines them using LLM-based reasoning to address shallow inter-modal reasoning by capturing semantic inconsistencies. ConLLM demonstrates strong performance across audio, video, and audio-visual modalities. It reduces audio deepfake EER by up to 50%, improves video accuracy by up to 8%, and achieves approximately 9% accuracy gains in audio-visual tasks. Ablation studies confirm that PTM-based embeddings contribute 9%-10% consistent improvements across modalities.

Figures (4)

• Figure 1: Illustration of common failure modes in multimodal deepfake detection. Top: Inconsistent unimodal predictions arise from poorly aligned audio-visual features (modality fragmentation). Bottom: Surface-level fusion leads to incorrect predictions due to the model’s inability to reason over subtle cross-modal inconsistencies (shallow inter-modal reasoning).
• Figure 2: Overview of the ConLLM architecture. Modality-specific embeddings are extracted using PTMs (XLS-R, VideoMAE, VATLM) and projected into a shared latent space. A contrastive learning block aligns embeddings, followed by LLM-based refinement using a transformer. Refined embeddings are concatenated and passed through a classifier to detect deepfake content.
• Figure 3: Heatmap of Equal Error Rate (EER %) across different PTMs and datasets
• Figure 4: Ablations to evaluate the impact of (i) modality-specific embedding extraction using PTMs, (ii) contrastive learning for cross-modal alignment, (iii) embedding refinement via LLMs, and (iv) comparison to alternative fusion strategies