Stacking Brick by Brick: Aligned Feature Isolation for Incremental Face Forgery Detection

TL;DR

This paper tackles the problem of incremental face forgery detection (IFFD) in the face of rapidly evolving forgery techniques. It introduces aligned feature isolation via Sparse Uniform Replay (SUR) and Latent-space Incremental Detector (LID) to stack task distributions brick by brick in the latent space, mitigating catastrophic forgetting and preserving forgery-specific information. SUR provides uniformly sampled replay data that approximate the previous global distribution, while LID performs feature isolation with distribution refilling and incremental decision alignment to fuse multi-task knowledge for robust binary detection. Extensive experiments on a comprehensive IFFD benchmark show that SUR-LID outperforms existing replay and regularization methods, with strong ablations, ablation-supported contributions, cross-dataset generalization, and robustness analyses, highlighting its practical impact for adaptive and scalable forgery detection systems.

Abstract

The rapid advancement of face forgery techniques has introduced a growing variety of forgeries. Incremental Face Forgery Detection (IFFD), involving gradually adding new forgery data to fine-tune the previously trained model, has been introduced as a promising strategy to deal with evolving forgery methods. However, a naively trained IFFD model is prone to catastrophic forgetting when new forgeries are integrated, as treating all forgeries as a single ''Fake" class in the Real/Fake classification can cause different forgery types overriding one another, thereby resulting in the forgetting of unique characteristics from earlier tasks and limiting the model's effectiveness in learning forgery specificity and generality. In this paper, we propose to stack the latent feature distributions of previous and new tasks brick by brick, $\textit{i.e.}$, achieving $\textbf{aligned feature isolation}$. In this manner, we aim to preserve learned forgery information and accumulate new knowledge by minimizing distribution overriding, thereby mitigating catastrophic forgetting. To achieve this, we first introduce Sparse Uniform Replay (SUR) to obtain the representative subsets that could be treated as the uniformly sparse versions of the previous global distributions. We then propose a Latent-space Incremental Detector (LID) that leverages SUR data to isolate and align distributions. For evaluation, we construct a more advanced and comprehensive benchmark tailored for IFFD. The leading experimental results validate the superiority of our method.

Figures (9)

• Figure 1: Illustration of the proposed aligned feature isolation in the latent space. Previous approaches (top) typically treat all forgeries, both old and new, as a single "Fake" class during incremental learning, causing feature distributions to override each other and limiting their ability to learn forgery specificity and generality. In contrast, we (bottom) propose incrementally adding new task distributions with isolation and alignment, akin to stacking new tasks "brick by brick" to the previous ones in the latent space. See Fig. \ref{['fig:Umap-dist']} for the experimental results of latent space distribution.
• Figure 2: Overall framework of the proposed method.
• Figure 3: Illustration of different replay strategies. Using Center coreddmp or Center and Hard dfil cannot preserve the global feature distribution, while the proposed SUR could uniformly sample a sparse version of the original global distribution.
• Figure 4: UMAP umap latent-space visualization for IFFD with Protocol 1. The upper row is the results of the baseline method (DFIL dfil) while the lower row is Ours. All shapes in blue are added for better illustration. The dashed lines denote the aligned boundary that divides real and fake. The dotted circles contain the distributions of newly incremented tasks.
• Figure 5: Evaluations of global distinction between the replay set and the training set. Maximum Mean Discrepancy (MMD) between different replay sets and their corresponding original training sets is deployed as the evaluation metric. A lower MMD indicates a smaller distinction between the replay set and the training set.