Adapter Merging with Centroid Prototype Mapping for Scalable Class-Incremental Learning

TL;DR

The paper tackles exemplar-free class-incremental learning by proposing ACMap, which merges per-task adapters into a single shared adapter to achieve constant inference time while preserving accuracy. It combines adapter merging to form a task-agnostic subspace with centroid prototype mapping to align and maintain previous representations, plus an early stopping strategy to control training cost. Key contributions include a simple average-based adapter merging with initial weight replacement, an affine centroid-based prototype mapping, and comprehensive experiments across five benchmarks showing competitive accuracy and significantly improved inference speed. The approach offers practical scalability for real-world sequential learning scenarios, with notable speedups over multi-adapter baselines and strong performance under domain shift, albeit with some VTAB-imbalance weaknesses and potential gains from an adapter-bank extension.

Abstract

We propose Adapter Merging with Centroid Prototype Mapping (ACMap), an exemplar-free framework for class-incremental learning (CIL) that addresses both catastrophic forgetting and scalability. While existing methods involve a trade-off between inference time and accuracy, ACMap consolidates task-specific adapters into a single adapter, thus achieving constant inference time across tasks without sacrificing accuracy. The framework employs adapter merging to build a shared subspace that aligns task representations and mitigates forgetting, while centroid prototype mapping maintains high accuracy by consistently adapting representations within the shared subspace. To further improve scalability, an early stopping strategy limits adapter merging as tasks increase. Extensive experiments on five benchmark datasets demonstrate that ACMap matches state-of-the-art accuracy while maintaining inference time comparable to the fastest existing methods. The code is available at https://github.com/tf63/ACMap.

Figures (15)

• Figure 1: Comparison of the final top-1 accuracy and inference time for task 40 of ImageNet-R in class-incremental learning. The comparison includes L2P L2P, DualPrompt DualPrompt, CODA-Prompt CODA-Prompt, SimpleCIL simplecil, APER simplecil, EASE ease, and our method. All methods use the same backbone (ViT-B/16). Our method performs well in terms of both inference time and accuracy by consolidating task-specific adapters into a single adapter.
• Figure 2: An Illustration of ACMap. ACMap sequentially trains an adapter for each task, starting from shared initial weights and incrementally merging them into a single adapter. In the subspace formed by the merged adapter, the prototypes for the current task are computed, while previous prototypes are updated via centroid prototype mapping.
• Figure 3: Visualization of the test error using linearly interpolated adapter weights $\bm \theta = u \bm \theta_{t-1} + v \bm \theta_t + (1 - u - v) \bm \theta_{t+1}, \,\,(0 \leq u,v \leq 1)$ across three consecutive adapter weights $\bm \theta_{t-1}, \bm \theta_t, \bm \theta_{t+1}$. Test errors for the adapters $\bm \theta_2, \bm \theta_3, \bm \theta_4$ are shown on the left, and for the adapters $\bm \theta_5, \bm \theta_6, \bm \theta_7$ on the right. The star symbol indicates the average merging ($u = 1\slash 3, v = 1\slash 3$). Additional results are provided in \ref{['sec:ap-landscape']}.
• Figure 4: The curve showing the differences in cosine similarity that arise when earlier task prototypes are substituted for prototypes in subsequent subspaces.
• Figure 5: Top-1 accuracy curve during CIL, comparing prototype-based methods: SimpleCIL (denoted as Simple), APER, and EASE. Additional results are provided in \ref{['sec:ap-full-results']}.