One Adapter for All: Towards Unified Representation in Step-Imbalanced Class-Incremental Learning

TL;DR

One-A is proposed, a unified and imbalance-aware framework that incrementally merges task updates into a single adapter, maintaining constant inference cost and demonstrating that a single, asymmetrically fused adapter can remain both adaptive to dynamic task sizes and efficient at deployment.

Abstract

Class-incremental learning (CIL) aims to acquire new classes over time while retaining prior knowledge, yet most setups and methods assume balanced task streams. In practice, the number of classes per task often varies significantly. We refer to this as step imbalance, where large tasks that contain more classes dominate learning and small tasks inject unstable updates. Existing CIL methods assume balanced tasks and therefore treat all tasks uniformly, producing imbalanced updates that degrade overall learning performance. To address this challenge, we propose One-A, a unified and imbalance-aware framework that incrementally merges task updates into a single adapter, maintaining constant inference cost. One-A performs asymmetric subspace alignment to preserve dominant subspaces learned from large tasks while constraining low-information updates within them. An information-adaptive weighting balances the contribution between base and new adapters, and a directional gating mechanism selectively fuses updates along each singular direction, maintaining stability in head directions and plasticity in tail ones. Across multiple benchmarks and step-imbalanced streams, One-A achieves competitive accuracy with significantly low inference overhead, showing that a single, asymmetrically fused adapter can remain both adaptive to dynamic task sizes and efficient at deployment.

Figures (12)

• Figure 1: Necessity of step-imbalanced continual learning and overview of our method. (a) Task splitting increases incremental steps, resulting in higher computation, more parameters, and stronger interference, which harms accuracy. (b) Direct merging averages task-specific parameters without alignment. (c) Our method aligns task subspaces via SVD and performs direction-aware fusion to preserve dominant knowledge while integrating information into a unified adapter with balanced stability and plasticity.
• Figure 2: Performance comparison under balanced and step-imbalanced settings. All methods show clear degradation under step-imbalanced tasks.
• Figure 3: Overview of One-A. For each layer, we decompose the large-task adapter via SVD to extract its dominant subspace $U_b\Sigma_b$ and project the small-task adapter for subspace alignment ($V_{a\rightarrow b}$). A global fusion is first performed between aligned components to obtain $V_{\mathrm{fused}}$, followed by direction-wise gating $g_i$ on the update $(V_{\mathrm{fused}} - V_b)$, which adaptively modulates each singular direction to balance stability and plasticity and yields the final merged adapter $\Delta W_{m}$.
• Figure 4: Accuracy at each step on ImageNet-A under different step-imbalance ratios $\gamma \in \{0.001, 0.005, 0.01, 0.05\}$.
• Figure 5: Forgetting curves under different configurations, along with the corresponding $\bar{A}$.