Fusion-SSAT: Unleashing the Potential of Self-supervised Auxiliary Task by Feature Fusion for Generalized Deepfake Detection

TL;DR

The paper tackles the limited cross-domain generalization of deepfake detectors by integrating a self-supervised local texture reconstruction task with RGB-based classification. Fusion-SSAT uses a shared ViT encoder to extract $f(R(v))$ for RGB and $f'(\tilde{L}(v))$ for masked LDP streams, fusing them via $z = f'(\tilde{L}(v)) \odot f(R(v))$ for classification, while a shallow decoder $g$ reconstructs masked LDP patches; training optimizes $L = \lambda L_{cls} + (1-\lambda) L_{rec}$ with $\lambda=0.1$. Experiments on FF++, Celeb-DF, FaceShifter, UADFV, DFD, and the large-scale DF40 dataset show that Fusion-SSAT achieves superior within-domain and cross-domain performance and stronger cross-forgery generalization than state-of-the-art detectors, with ablations confirming the benefit of combining local texture cues with global RGB representations. The results suggest that leveraging local texture patterns via SSL alongside global semantic cues yields more robust and transferable deepfake detectors for real-world deployment.

Abstract

In this work, we attempted to unleash the potential of self-supervised learning as an auxiliary task that can optimise the primary task of generalised deepfake detection. To explore this, we examined different combinations of the training schemes for these tasks that can be most effective. Our findings reveal that fusing the feature representation from self-supervised auxiliary tasks is a powerful feature representation for the problem at hand. Such a representation can leverage the ultimate potential and bring in a unique representation of both the self-supervised and primary tasks, achieving better performance for the primary task. We experimented on a large set of datasets, which includes DF40, FaceForensics++, Celeb-DF, DFD, FaceShifter, UADFV, and our results showed better generalizability on cross-dataset evaluation when compared with current state-of-the-art detectors.

Figures (5)

• Figure 1: Overview of the self-supervised auxiliary task (SSAT) framework.
• Figure 2: Overview of the proposed Fusion-SSAT approach
• Figure 3: ROCs for in-domain and cross-domain evaluations.
• Figure 4: ROCs of cross-forgery evaluations on DF40 dataset.
• Figure 5: ROCs of cross-domain evaluations on the DF40 dataset.