Continual Adapter Tuning with Semantic Shift Compensation for Class-Incremental Learning

TL;DR

This work tackles class-incremental learning with pre-trained Vision Transformers by showing that incrementally tuning a shared adapter outperforms prompt-based PET methods and avoiding parameter constraints enhances plasticity. It introduces a two-stage approach: (1) continuous adapter fine-tuning and local classifier updates, and (2) retraining a unified classifier using Gaussian-feature sampling and semantic-shift compensation for old prototypes without past data. The key contributions include eliminating adapter pools and data retention, achieving state-of-the-art results on multiple CIL benchmarks, and extending the approach to FSCIL and HCIL. Overall, the method offers a cost-efficient, data-sparing path to robust continual learning with strong generalization across domains.

Abstract

Class-incremental learning (CIL) aims to enable models to continuously learn new classes while overcoming catastrophic forgetting. The introduction of pre-trained models has brought new tuning paradigms to CIL. In this paper, we revisit different parameter-efficient tuning (PET) methods within the context of continual learning. We observe that adapter tuning demonstrates superiority over prompt-based methods, even without parameter expansion in each learning session. Motivated by this, we propose incrementally tuning the shared adapter without imposing parameter update constraints, enhancing the learning capacity of the backbone. Additionally, we employ feature sampling from stored prototypes to retrain a unified classifier, further improving its performance. We estimate the semantic shift of old prototypes without access to past samples and update stored prototypes session by session. Our proposed method eliminates model expansion and avoids retaining any image samples. It surpasses previous pre-trained model-based CIL methods and demonstrates remarkable continual learning capabilities. Experimental results on five CIL benchmarks validate the effectiveness of our approach, achieving state-of-the-art (SOTA) performance.

Figures (10)

• Figure 1: Comparison of different parameter-efficient tuning CIL baselines on CIFAR100 dataset. Left: The relationship between the average accuracy of the incremental sessions and the number of tunable parameters. Right: The average performance of old classes and new classes for each PET method. This figure show that Adapter tuning performs best and is more stable across datasets compared to other PET methods.
• Figure 2: The framework of our proposed method. Left: The illustration of the structure of ViT and adapter. The adapter and local classifier are incrementally trained in each session using the Eq. \ref{['loss']}. Right: The process of retraining the classifier with semantic shift estimation.
• Figure 3: Comparison of the performance on ImageNetR dataset with different extent of parameter constraints. Left: The overall accuracy of each session. Right: The accuracy of new classes.
• Figure 4: Parameter sensitivity analysis on the ImageNetR dataset. Left: The parameter sensitiveness of two incremental tasks. Right: The sensitiveness of different parameters in one task.
• Figure 5: The performance of each learning session on six CIL datasets. (a) ImageNetR; (b) ImageNetA; (c) CUB200; (d) CIFAR100; (e) Stanford Cars; (f) Food-101 . These curves are plotted by calculating the average performance across three different seeds for each incremental session.