Zero-Shot Fake Video Detection by Audio-Visual Consistency

TL;DR

The paper tackles the problem of generalizing fake video detection to unseen deepfakes without using fake data. It proposes a zero-shot framework with a two-layer pipeline: a frontend that encodes audio-visual content with pre-trained cross-modal models, and a backend that measures audio-visual consistency, including a novel content-consistency detector (CCFD) that uses ASR and VSR to compare decoded content via Word Error Rate. Three detectors—SCFD (semantic), TCFD (temporal), and CCFD (content)—are evaluated on FakeAVCeleb and DeepFakeTIMIT, and a simple fusion of their scores yields state-of-the-art results and improved robustness to perturbations. The approach demonstrates strong generalization across diverse deepfake techniques and perturbations, with practical impact for real-world audio-visual forgery detection that relies only on genuine data for modelling.

Abstract

Recent studies have advocated the detection of fake videos as a one-class detection task, predicated on the hypothesis that the consistency between audio and visual modalities of genuine data is more significant than that of fake data. This methodology, which solely relies on genuine audio-visual data while negating the need for forged counterparts, is thus delineated as a `zero-shot' detection paradigm. This paper introduces a novel zero-shot detection approach anchored in content consistency across audio and video. By employing pre-trained ASR and VSR models, we recognize the audio and video content sequences, respectively. Then, the edit distance between the two sequences is computed to assess whether the claimed video is genuine. Experimental results indicate that, compared to two mainstream approaches based on semantic consistency and temporal consistency, our approach achieves superior generalizability across various deepfake techniques and demonstrates strong robustness against audio-visual perturbations. Finally, state-of-the-art performance gains can be achieved by simply integrating the decision scores of these three systems.

Figures (2)

• Figure 1: The unified framework of zero-shot fake video detection.
• Figure 2: AUC on FakeAVCeleb under levels of perturbations.