Task-Agnostic Guided Feature Expansion for Class-Incremental Learning

TL;DR

Class-incremental learning suffers from feature collision when new task features interfere with old ones. TagFex introduces a Task-Agnostic Guided Feature Expansion framework that continually acquires task-agnostic features in a separate model, merges them with task-specific features via a merge attention module, and transfers the enriched representation back to the task-specific classifiers using a $D_{KL}$-based knowledge transfer. This yields more diverse task representations, reduces collision, and delivers state-of-the-art results on CIFAR100 and ImageNet100 across several CIL setups, with a pruning variant TagFex-P offering memory efficiency. The results validate that task-agnostic information can be effectively exploited to enhance incremental learning while maintaining competitive parameter budgets.

Abstract

The ability to learn new concepts while preserve the learned knowledge is desirable for learning systems in Class-Incremental Learning (CIL). Recently, feature expansion of the model become a prevalent solution for CIL, where the old features are fixed during the training of the new task while new features are expanded for the new tasks. However, such task-specific features learned from the new task may collide with the old features, leading to misclassification between tasks. Therefore, the expanded model is often encouraged to capture diverse features from the new task, aiming to avoid such collision. However, the existing solution is largely restricted to the samples from the current task, because of the poor accessibility to previous samples. To promote the learning and transferring of diverse features across tasks, we propose a framework called Task-Agnostic Guided Feature Expansion (TagFex). Firstly, it captures task-agnostic features continually with a separate model, providing extra task-agnostic features for subsequent tasks. Secondly, to obtain useful features from the task-agnostic model for the current task, it aggregates the task-agnostic features with the task-specific feature using a merge attention. Then the aggregated feature is transferred back into the task-specific feature for inference, helping the task-specific model capture diverse features. Extensive experiments show the effectiveness and superiority of TagFex on various CIL settings. Code is available at https://github.com/bwnzheng/TagFex_CVPR2025.

Figures (34)

• Figure 1: Average CKA feature similarities between different expanded models of DER and TagFex by task. It shows the expanded features learned by TagFex are less correlated.
• Figure 2: GradCAM visualization comparison between different expanded models by DER and TagFex. It shows that the expanded models learned by the proposed framework focus on more diverse features.
• Figure 3: Feature Expansion Without Rehearsal Samples
• Figure 4: Feature Expansion with More Diverse Features
• Figure 6: Overview of the proposed framework TagFex. The framework is composed of three parts: Task-agnostic Model (Section \ref{['sec:acq_ta']}), Merge Attention (Section \ref{['sec:merge_attn']}) and Task-specific Models (Section \ref{['sec:ts_expansion']} and \ref{['sec:ts_trans']}). Overall, our framework continually captures the task-agnostic feature in each task and transfers such feature back to the task-specific feature, helping the task-specific model obtain more diverse features to combat the feature collision.