Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Need

TL;DR

The paper rethinks class-incremental learning in the era of pre-trained models, arguing that generalizability of PTMs and adaptivity to downstream data are both essential. It introduces Aper, a unified framework that adapts the PTM in the first stage and then merges the adapted and original embeddings to form a robust, prototype-based classifier, maintaining generalizability while enabling task-specific adaptation. Through extensive experiments on seven benchmarks, including four new domain-gap datasets (ImageNet-A, ObjectNet, OmniBenchmark, VTAB), Aper consistently outperforms state-of-the-art PTM-based CIL methods, while remaining parameter-efficient. The work also provides ablations, visualizations, and cross-domain evidence to validate the benefits of the adapt-then-merge approach and the importance of limiting adaptation to the initial incremental stage.

Abstract

Class-incremental learning (CIL) aims to adapt to emerging new classes without forgetting old ones. Traditional CIL models are trained from scratch to continually acquire knowledge as data evolves. Recently, pre-training has achieved substantial progress, making vast pre-trained models (PTMs) accessible for CIL. Contrary to traditional methods, PTMs possess generalizable embeddings, which can be easily transferred for CIL. In this work, we revisit CIL with PTMs and argue that the core factors in CIL are adaptivity for model updating and generalizability for knowledge transferring. 1) We first reveal that frozen PTM can already provide generalizable embeddings for CIL. Surprisingly, a simple baseline (SimpleCIL) which continually sets the classifiers of PTM to prototype features can beat state-of-the-art even without training on the downstream task. 2) Due to the distribution gap between pre-trained and downstream datasets, PTM can be further cultivated with adaptivity via model adaptation. We propose AdaPt and mERge (APER), which aggregates the embeddings of PTM and adapted models for classifier construction. APER is a general framework that can be orthogonally combined with any parameter-efficient tuning method, which holds the advantages of PTM's generalizability and adapted model's adaptivity. 3) Additionally, considering previous ImageNet-based benchmarks are unsuitable in the era of PTM due to data overlapping, we propose four new benchmarks for assessment, namely ImageNet-A, ObjectNet, OmniBenchmark, and VTAB. Extensive experiments validate the effectiveness of APER with a unified and concise framework. Code is available at https://github.com/zhoudw-zdw/RevisitingCIL

Figures (7)

• Figure 1: Comparison of different PTM-based CIL methods on VTAB dataset. The X-axis stands for the number of tunable parameters, and the Y-axis represents the average accuracy. The radius stands for the training time. Although consuming more tuning parameters and training time, current state-of-the-art (i.e., L2P and DualPrompt) still show inferior performance than the baseline method SimpleCIL. By contrast, our Aper consistently improves the baseline with tiny costs. For a fair comparison, all methods are based on pre-trained ViT-B/16-IN1K. SimpleCIL utilizes the training set to calculate the average embeddings.
• Figure 2: Performance of new and old classes in CIL with PTM. Sequentially finetuning the model fills the domain gap and performs better on new classes, while freezing the model has better generalizability and performs better on old classes.
• Figure 3: Illustration of Aper. Left: the training protocol of Aper. We adapt the PTM using the first stage training set $\mathcal{D}^1$ and then concatenate the embedding functions of PTM and the adapted model to maintain generalizability and adaptivity. The aggregated embedding function $[\phi^*(\cdot),\phi(\cdot)]$ is frozen throughout the following stages, and we extract the prototypes via Eq. \ref{['eq:prototype-ours']} to set the classifier. Middle: adapting pre-trained ViT for CIL. We provide VPT Deep/Shallow, Scale & Shift, and Adapter for model adaptation. Right: adapting pre-trained CNN for CIL. We provide BN tuning and Scale & Shift for model adaptation. Aper is a general framework that can be orthogonally combined with these adapting techniques. Red modules in the figure are trainable, while gray ones are frozen.
• Figure 4: (a)$\sim$(f): Incremental performance with ViT-B/16-IN1K as the backbone when half of the total classes are base classes. (g)$\sim$(i): Incremental performance when using ResNet18 as backbone. Since L2P and Dualprompt cannot be deployed with ResNet, we do not report their performance in (g)$\sim$(i). Aper consistently improves the performance of different backbones, i.e., ViT and CNN.
• Figure 5: Ablation study. (a)-(c): We use PCA or random sample to downscale the dimension of aggregated embeddings. (d): We compare Aper to its sub-modules for ablation. (e): Number of total parameters of different compared methods. The bars with shadow denote the parameters used during training but dropped during inference. (f): The accuracy trend with the change of adapting stages. Adapt-$T$ denotes the model is adapted for the first $T$ incremental tasks. $T=0$ denotes SimpleCIL.