Investigating self-supervised representations for audio-visual deepfake detection

TL;DR

This work systematically benchmarks a wide range of self-supervised representations for audio-visual deepfake detection, examining their detection power, interpretability, and cross-modal complementarity. By using linear probing and two anomaly-detection proxies, the study reveals that many SSL features encode meaningful deepfake cues and that their information is often complementary across modalities. Temporal and spatial explanations show that models attend to semantically relevant regions, with audio models leveraging cues like leading silence. However, strong in-domain performance does not generalize reliably across datasets, highlighting a fundamental generalization gap that goes beyond the SSL features themselves and indicating a need for robust cross-domain strategies and improved evaluation protocols.

Abstract

Self-supervised representations excel at many vision and speech tasks, but their potential for audio-visual deepfake detection remains underexplored. Unlike prior work that uses these features in isolation or buried within complex architectures, we systematically evaluate them across modalities (audio, video, multimodal) and domains (lip movements, generic visual content). We assess three key dimensions: detection effectiveness, interpretability of encoded information, and cross-modal complementarity. We find that most self-supervised features capture deepfake-relevant information, and that this information is complementary. Moreover, models primarily attend to semantically meaningful regions rather than spurious artifacts. Yet none generalize reliably across datasets. This generalization failure likely stems from dataset characteristics, not from the features themselves latching onto superficial patterns. These results expose both the promise and fundamental challenges of self-supervised representations for deepfake detection: while they learn meaningful patterns, achieving robust cross-domain performance remains elusive.

Figures (7)

• Figure 1: We evaluate a wide array of self-supervised representations for audio-visual deepfake detection using a multi-faceted evaluation addressing three research questions: their usefulness and robustness (via classification with linear probing and anomaly detection), their focus and interpretability (through temporal and spatial explanations), and their complementarity (via correlation and fusion analyses).
• Figure 2: Performance of temporal localization of explanations (solid bars) and how it compares to the deepfake classification (hatched bars). Color indicates modality.
• Figure 3: Temporal explanations of the top video predictions for four SSL representations. The predictions are given in terms of unnormalized scores (logits) and probabilities. Red regions indicate fake segments, and gray dashed lines correspond to the decision boundary (0.5 probability). For audio models we show Mel spectrograms; for vision models we show three frames (corresponding to the triangle markers on the line plot).
• Figure 4: Alignment of spatial explanations to human annotations. Left: Alignment error in terms of mean absolute error (MAE) as a function of the model confidence (fakeness score). The explanations align better to human annotations as the model is more confident in its predictions. Right: Qualitative samples human annotation shown as the center of the red circle on top frame, and explanation of the CLIP-based model shown on bottom frame (maximum value indicated by the green circle).
• Figure 5: Correlations between models (left) and downstream performance (right). The downstream performance is presented in absolute values for the unimodal models (AUC column) and as relative improvement for feature combinations. Training was done on AV1M, testing on FAVC.