Transcending Forgery Specificity with Latent Space Augmentation for Generalizable Deepfake Detection

TL;DR

This work tackles the generalization gap in deepfake detection by addressing forgery-specific overfitting. It introduces LSDA, a latent-space data augmentation framework that enlarges the forgery space through within-domain transformations (centrifugal, affine, additive) and cross-domain Mixup, coupled with a teacher-student distillation architecture and an ArcFace-informed real-feature prior. The method yields a generalizable detector by distilling augmented forgery representations into a single student encoder with a binary classifier, optimized via Domain, Distillation, and Binary losses. Extensive experiments across FF++ and unseen datasets show improved cross-dataset performance and robustness, outperforming state-of-the-art methods and RGB-based augmentations, with insightful visualizations corroborating the broader, more transferable detection capabilities. Overall, latent-space augmentation offers a scalable, less artifact-dependent path to robust deepfake detection.

Abstract

Deepfake detection faces a critical generalization hurdle, with performance deteriorating when there is a mismatch between the distributions of training and testing data. A broadly received explanation is the tendency of these detectors to be overfitted to forgery-specific artifacts, rather than learning features that are widely applicable across various forgeries. To address this issue, we propose a simple yet effective detector called LSDA (\underline{L}atent \underline{S}pace \underline{D}ata \underline{A}ugmentation), which is based on a heuristic idea: representations with a wider variety of forgeries should be able to learn a more generalizable decision boundary, thereby mitigating the overfitting of method-specific features (see Fig.~\ref{fig:toy}). Following this idea, we propose to enlarge the forgery space by constructing and simulating variations within and across forgery features in the latent space. This approach encompasses the acquisition of enriched, domain-specific features and the facilitation of smoother transitions between different forgery types, effectively bridging domain gaps. Our approach culminates in refining a binary classifier that leverages the distilled knowledge from the enhanced features, striving for a generalizable deepfake detector. Comprehensive experiments show that our proposed method is surprisingly effective and transcends state-of-the-art detectors across several widely used benchmarks.

Figures (6)

• Figure 1: Toy examples for intuitively illustrating our proposed latent space augmentation strategy. The baseline can be overfitted to forgery-specific features and thus cannot generalize well for unseen forgeries. In contrast, our proposed method avoids overfitting to specific forgery features by enlarging the forgery space through latent space augmentation. This approach aims to equip our method with the capability to effectively adjust and adapt to new and previously unseen forgeries.
• Figure 2: The overall pipeline of our proposed method (two fake types are considered as an example). (1) In the training phase, the student encoder is trained to learn a generalizable and robust feature by utilizing the distribution match to distill the knowledge of the real and fake teacher encoders to the student encoder. (2) In the inference phase, only the student encoder is applied to detect the fakes from the real. (3) For the learning of the forgery feature, we apply the latent space within-domain (WD) and cross-domain (CD) augmentation. (4) For the learning of the real feature, the pre-trained and frozen ArcFace face recognition model is applied. (5) WD involves novel augmentations to fine-tune domain-specific features, while CD enables the model to seamlessly identify transitions between different types of forgeries.
• Figure 3: Robustness to Unseen Perturbations: We report video-level AUC (%) under five different degradation levels of five specific types of perturbations jiang2020deeperforensics. We compare our results with three RGB-based augmentation-based methods to demonstrate our robustness. Best viewed in color.
• Figure 4: t-SNE visualization of latent space w and wo augmentations.
• Figure 5: GradCAM visualizations selvaraju2017grad for fake samples from different forgeries. We compare the baseline (EFNB4 tan2019efficientnet with ours. "Mask (GT)" highlights the ground truth of the manipulation region. Best viewed in color.