Semantic-Guided Dynamic Sparsification for Pre-Trained Model-based Class-Incremental Learning

TL;DR

This work reframes continual learning with pre-trained models by shifting from constraining adapter parameters to actively guiding activation subspaces. The proposed Semantic-Guided Dynamic Sparsification (SGDS) operates in two phases—Semantic Exploration to align subspace orientation and Activation Compaction to reduce subspace rank—producing sparse, task-specific activation streams that minimize interference while preserving plasticity. Across CIFAR-100, ImageNet-R, ImageNet-A, and ObjectNet, SGDS achieves state-of-the-art performance in exemplar-free settings, with ablations showing both semantic guidance and activation compaction are essential. The activation-centric approach offers strong scalability, privacy preservation, and practical robustness, making it a compelling alternative to parameter-space regularization for PTM-based class-incremental learning.

Abstract

Class-Incremental Learning (CIL) requires a model to continually learn new classes without forgetting old ones. A common and efficient solution freezes a pre-trained model and employs lightweight adapters, whose parameters are often forced to be orthogonal to prevent inter-task interference. However, we argue that this parameter-constraining method is detrimental to plasticity. To this end, we propose Semantic-Guided Dynamic Sparsification (SGDS), a novel method that proactively guides the activation space by governing the orientation and rank of its subspaces through targeted sparsification. Specifically, SGDS promotes knowledge transfer by encouraging similar classes to share a compact activation subspace, while simultaneously preventing interference by assigning non-overlapping activation subspaces to dissimilar classes. By sculpting class-specific sparse subspaces in the activation space, SGDS effectively mitigates interference without imposing rigid constraints on the parameter space. Extensive experiments on various benchmark datasets demonstrate the state-of-the-art performance of SGDS.

Figures (9)

• Figure 1: Comparison between parameter-constraining and our activation-guiding methods. (a) Constraining adapter parameters (e.g., via orthogonality $\perp$) to reduce interference often harms model plasticity. (b) Our method guides the adapter's input activations into orthogonal subspaces. This separation mitigates inter-task interference while preserving the model's plasticity, as its parameters remain unconstrained.
• Figure 2: An overview of SGDS. Operating at the input of each backbone adapter, SGDS proactively guides the activation space. The process begins by selecting a learning strategy based on semantic similarity: Knowledge Reuse (top path) to foster knowledge transfer, or New Subspace Allocation (bottom path) to prevent interference. This decision then guides the Semantic Exploration phase, which governs the subspace's orientation by either aligning it with an existing subspace or creating a new orthogonal one. Subsequently, the Activation Compaction phase reduces the subspace's rank by enforcing targeted sparsity, making it more compact. This entire process delivers a strategically structured activation stream to each adapter, freeing it from inter-task interference and thereby achieving a superior stability-plasticity balance.
• Figure 3: Performance comparison with rehearsal-based methods on CIFAR-100 (left), ImageNet-R (center), and ObjectNet (right). The plots track the average accuracy as the model learns from a sequence of tasks. SGDS consistently outperforms strong baselines, and the performance gap widens over time, indicating superior knowledge retention.
• Figure 4: t-SNE visualizations of parameter and activation distributions on ImageNet-R. Colors represent tasks, and numbers denote layers. (a) The down-projection parameters ($W_{\text{down}}$) cluster tightly by layer regardless of the task, indicating they learn a shared, task-agnostic function. (b) In contrast, the up-projection parameters ($W_{\text{up}}$) are dispersed and form task-specific clusters, revealing their role in specializing for each task. (c) Similarly, the input activations are intermingled in early layers but become clearly separated by task in later layers.
• Figure 5: Hyperparameter Robustness on ImageNet-R. The plots show the average incremental accuracy as a function of (a) the compaction strength $\gamma$ and exploration balance $\beta$, and (b) the sparsity ratio $k$. SGDS demonstrates stable performance across a wide range of values for all key hyperparameters.