Deepfake Detection with Multi-Artifact Subspace Fine-Tuning and Selective Layer Masking

TL;DR

This work tackles cross-dataset deepfake detection by explicitly decoupling semantic and artifact representations through SVD-based weight decomposition, freezing a semantic subspace while learning multiple artifact subspaces. It introduces a selective layer masking strategy that gates parameter updates based on gradient stability, reducing semantic drift during fine-tuning. Orthogonality and spectral consistency constraints further regularize the artifact subspaces, yielding robust cross-domain performance and resilience to common perturbations. The approach, validated on multiple benchmarks, demonstrates strong generalization without extra supervision and offers a scalable framework for modeling diverse forgery artifacts in real-world scenarios.

Abstract

Deepfake detection still faces significant challenges in cross-dataset and real-world complex scenarios. The root cause lies in the high diversity of artifact distributions introduced by different forgery methods, while pretrained models tend to disrupt their original general semantic structures when adapting to new artifacts. Existing approaches usually rely on indiscriminate global parameter updates or introduce additional supervision signals, making it difficult to effectively model diverse forgery artifacts while preserving semantic stability. To address these issues, this paper proposes a deepfake detection method based on Multi-Artifact Subspaces and selective layer masks (MASM), which explicitly decouples semantic representations from artifact representations and constrains the fitting strength of artifact subspaces, thereby improving generalization robustness in cross-dataset scenarios. Specifically, MASM applies singular value decomposition to model weights, partitioning pretrained weights into a stable semantic principal subspace and multiple learnable artifact subspaces. This design enables decoupled modeling of different forgery artifact patterns while preserving the general semantic subspace. On this basis, a selective layer mask strategy is introduced to adaptively regulate the update behavior of corresponding network layers according to the learning state of each artifact subspace, suppressing overfitting to any single forgery characteristic. Furthermore, orthogonality constraints and spectral consistency constraints are imposed to jointly regularize multiple artifact subspaces, guiding them to learn complementary and diverse artifact representations while maintaining a stable overall spectral structure.

Figures (3)

• Figure 1: A simple illustrative example is provided to intuitively demonstrate the proposed deepfake detection method based on multi-artifact subspace fine-tuning and selective layer masking. The baseline method learns within a single feature space, which limits the diversity of forgery patterns it can capture and makes it prone to overfitting to specific forgery types during training, thereby weakening its generalization to unseen forgeries. In contrast, the proposed method partitions the forgery feature space into multiple artifact subspaces to model more diverse forgery types and adaptively controls weight updates according to the learning state of each subspace, effectively alleviating overfitting.
• Figure 2: Overall overview of the proposed MASM framework.(a) Training pipeline. Starting from the original model, the model is iteratively optimized by introducing the Multi-Artifact Subspace Fine-Tuning (MASFT) mechanism, together with a selective layer masking strategy that progressively updates key layer parameters, resulting in the final discriminative model.(b) Selective layer masking. The bias–variance ratio is computed for each network layer, and the Top-K layers are adaptively selected based on the ranking results to construct a binary layer mask that guides the subsequent fine-tuning process.(c) Multi-artifact subspace fine-tuning. Model weights are decomposed into a principal subspace and artifact-related subspaces via singular value decomposition (SVD), enabling targeted fine-tuning within multiple artifact subspaces.(d) Multi-artifact subspace constraints. Orthogonality and spectral consistency constraints are imposed on the left singular vectors, singular values, and right singular vectors to enhance the decoupling among different artifact subspaces.
• Figure 3: Robustness evaluation results.We compare our method with FA Cui2025ForensicsAdapter, ProDet cheng2024ProDet, and Effort Yan2024OrthogonalSD in terms of video-level AUC under six types of distortions and five different intensity levels. It can be observed that MASM maintains stable and consistently high detection performance across different distortion types and severity levels.