Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learning

TL;DR

This work tackles class-incremental learning (CIL) with pre-trained models by addressing catastrophic forgetting in exemplar-free settings. It introduces ExpAndable Subspace Ensemble (EASE), which builds multiple task-specific subspaces via lightweight adapters frozen on the PTM, concatenates the resulting embeddings, and classifies with a prototype-based, reweighted ensemble. A semantic-guided prototype complement synthesizes old-class prototypes in new subspaces without exemplars by leveraging cross-task feature relationships in a co-occurrence space, enabling alignment across expanding feature spaces. Empirically, EASE achieves state-of-the-art performance on seven benchmarks with competitive parameter costs, demonstrating strong scalability, efficiency, and robustness for PTM-based continual learning. The approach advances practical continual learning by combining low-cost subspace expansion with exemplar-free classifier completion, making it suitable for real-world data streams.

Abstract

Class-Incremental Learning (CIL) requires a learning system to continually learn new classes without forgetting. Despite the strong performance of Pre-Trained Models (PTMs) in CIL, a critical issue persists: learning new classes often results in the overwriting of old ones. Excessive modification of the network causes forgetting, while minimal adjustments lead to an inadequate fit for new classes. As a result, it is desired to figure out a way of efficient model updating without harming former knowledge. In this paper, we propose ExpAndable Subspace Ensemble (EASE) for PTM-based CIL. To enable model updating without conflict, we train a distinct lightweight adapter module for each new task, aiming to create task-specific subspaces. These adapters span a high-dimensional feature space, enabling joint decision-making across multiple subspaces. As data evolves, the expanding subspaces render the old class classifiers incompatible with new-stage spaces. Correspondingly, we design a semantic-guided prototype complement strategy that synthesizes old classes' new features without using any old class instance. Extensive experiments on seven benchmark datasets verify EASE's state-of-the-art performance. Code is available at: https://github.com/sun-hailong/CVPR24-Ease

Figures (12)

• Figure 1: Parameter-performance comparison of different methods on ImageNet-R B100 Inc50. All methods utilize the same PTM as initialization. Ease requires the same scale parameters as other prompt-based methods wang2022dualpromptsmith2023codawang2022learning while performing best among all competitors without using exemplars.
• Figure 1: Experimental results on different similarity calculation methods. Using prototype-prototype similarity shows better performance than prototype-instance similarity.
• Figure 2: Illustration of Ease. Left: In the first task, we learn an adapter $\mathcal{A}_1$ to encode task specific features, and extract class prototypes $\mathbf{P}_{1,1}$. Middle: In the second task, we initialize a new adapter $\mathcal{A}_2$ to encode new features, and extract prototypes $\mathbf{P}_{2,1}$ and $\mathbf{P}_{2,2}$. Without exemplars, we need to synthesize $\mathbf{P}_{1,2}$ (old class prototypes in the new subspace) for prediction. Right: Semantic mapping process. We extract class-wise similarity in the co-occurrence subspace and utilize it to synthesize old class prototypes in the target space.
• Figure 2: Experimental results on different subspace tuning methods. Using adapter tuning shows better performance than VPT.
• Figure 3: Performance curve of different methods under different settings. All methods are initialized with ViT-B/16-IN1K. We annotate the relative improvement of Ease above the runner-up method with numerical numbers at the last incremental stage.