A Multi-Stream Fusion Approach with One-Class Learning for Audio-Visual Deepfake Detection

TL;DR

The paper tackles the challenge of detecting audio-visual deepfakes that generalize to unseen generation methods while providing modality-level interpretability. It introduces MSOC, a multi-stream architecture with three modality-specific branches trained via one-class losses and fused at inference, extending OC-Softmax to the AV setting. A new dataset split based on generation methods creates four test sets (RAFV, FAFV, FARV, Unsynced) to rigorously evaluate unseen attacks and synchronization cues, and results show MSOC achieves strong generalization and interpretability compared to state-of-the-art baselines. The work demonstrates that modality-specific one-class learning plus targeted fusion improves robustness to unseen fakes and offers insight into which modality drives the decision, with practical implications for reliable AV deepfake detection and transparent AI systems.

Abstract

This paper addresses the challenge of developing a robust audio-visual deepfake detection model. In practical use cases, new generation algorithms are continually emerging, and these algorithms are not encountered during the development of detection methods. This calls for the generalization ability of the method. Additionally, to ensure the credibility of detection methods, it is beneficial for the model to interpret which cues from the video indicate it is fake. Motivated by these considerations, we then propose a multi-stream fusion approach with one-class learning as a representation-level regularization technique. We study the generalization problem of audio-visual deepfake detection by creating a new benchmark by extending and re-splitting the existing FakeAVCeleb dataset. The benchmark contains four categories of fake videos (Real Audio-Fake Visual, Fake Audio-Fake Visual, Fake Audio-Real Visual, and Unsynchronized videos). The experimental results demonstrate that our approach surpasses the previous models by a large margin. Furthermore, our proposed framework offers interpretability, indicating which modality the model identifies as more likely to be fake. The source code is released at https://github.com/bok-bok/MSOC.

Figures (4)

• Figure 1: Overview of Multi-Stream Architecture. In the figure, black dashed lines represent the training process, and solid purple lines represent the inference process. $\bigoplus$ symbol represents feature concatenation. + symbol means addition.
• Figure 2: Score distribution visualization for four fake categories across each MSOC branch. Scores close to 1 indicate the model perceives the modality as real, and the red vertical line denotes the decision threshold.
• Figure 3: t-SNE visualization of concatenated audio-visual feature. The cross "X" in the figure represents the center of the data for each category. Better viewed in color.
• Figure 4: The figure compares visual scores computed from ResNet and SCNet-STIL visual feature extractors on both fake and real samples.