A Unified Framework for Modality-Agnostic Deepfakes Detection

TL;DR

A comprehensive framework that is agnostic to fake modalities is introduced, which facilitates the identification of multimodal deepfakes and handles situations with missing modalities, regardless of the manipulations embedded in audio, video, or even cross-modal forms.

Abstract

As AI-generated content (AIGC) thrives, deepfakes have expanded from single-modality falsification to cross-modal fake content creation, where either audio or visual components can be manipulated. While using two unimodal detectors can detect audio-visual deepfakes, cross-modal forgery clues could be overlooked. Existing multimodal deepfake detection methods typically establish correspondence between the audio and visual modalities for binary real/fake classification, and require the co-occurrence of both modalities. However, in real-world multi-modal applications, missing modality scenarios may occur where either modality is unavailable. In such cases, audio-visual detection methods are less practical than two independent unimodal methods. Consequently, the detector can not always obtain the number or type of manipulated modalities beforehand, necessitating a fake-modality-agnostic audio-visual detector. In this work, we introduce a comprehensive framework that is agnostic to fake modalities, which facilitates the identification of multimodal deepfakes and handles situations with missing modalities, regardless of the manipulations embedded in audio, video, or even cross-modal forms. To enhance the modeling of cross-modal forgery clues, we employ audio-visual speech recognition (AVSR) as a preliminary task. This efficiently extracts speech correlations across modalities, a feature challenging for deepfakes to replicate. Additionally, we propose a dual-label detection approach that follows the structure of AVSR to support the independent detection of each modality. Extensive experiments on three audio-visual datasets show that our scheme outperforms state-of-the-art detection methods with promising performance on modality-agnostic audio/video deepfakes.

Figures (5)

• Figure 1: Forgery patterns in deepfakes. (a) Visual artifacts within a single modality: Biden's teeth exhibit an irregular shape. (b) Cross-modal speech mismatch in deepfakes. The phoneme-level audio waveform corresponds to the duration of the frame shown above. Both images reveal a speech mismatch between lip motions and audio sounds. On the left, Biden's mouth is in a closed state before pronouncing the "f" sound and has not yet produced a complete syllable. However, the audio image displays a complex frequency wave that should only appear during active pronunciation. On the right, Biden is uttering the vowels in "you", but the sound waves show rather simple frequency patterns, which are typically indicative of panting or background noise rather than speech.
• Figure 2: Modality-Agnostic Audio-Visual Deepfake Detection: The figure on the left shows three modality-agnostic detection scenarios that our detector supports. This detector enables independent detection of forgery in each modality and can handle scenarios where either modality is unavailable. The figure on the right illustrates the general scheme of our method, which leverages the speech correlation across modalities via the AVSR task to perform dual-label detection.
• Figure 3: Demonstration of learning scheme of dual-label deepfake detection. (a) the overview of the proposed framework. The two encoders and joint decoder, which are pretrained on AVSR, are finetuned to model speech correlation across modalities. (b) Modality Compensation Adapter (MCA) is embedded into the encoder-decoder to prevent the network's over-reliance on a certain modality. (c) Dual-Label Classifier (DLC) is attached at the backend of the model to perform both audio and visual manipulation detection at once.
• Figure 4: The performance comparison on different types of fake videos on the FakeAVCeleb dataset.
• Figure 5: T-SNE visualization of feature space learned by variants of our framework.