DRL: Discriminative Representation Learning with Parallel Adapters for Class Incremental Learning

TL;DR

The paper tackles non-rehearsal Class-Incremental Learning by identifying three core challenges: escalating model complexity, non-smooth representation shifts across stages, and misalignment between stage-wise optimization and global inference. It introduces Discriminative Representation Learning (DRL), composed of Incremental Parallel Adapter (IPA) and Decoupled Anchor Supervision (DAS). IPA augments a fixed Pre-Trained Model with lightweight adapters and a learnable transfer gate to achieve smooth representation shifts with minimal parameters, while DAS uses a fixed virtual anchor $k$ to decouple positive and negative constraints, aligning feature spaces across stages. The approach, optimized with $L_{total} = L_{das} + \alpha L_{kd}$, delivers state-of-the-art results on six benchmarks, with strong robustness to distribution shifts and efficient single-pass inference, making it well-suited for privacy-preserving, streaming data settings.

Abstract

With the excellent representation capabilities of Pre-Trained Models (PTMs), remarkable progress has been made in non-rehearsal Class-Incremental Learning (CIL) research. However, it remains an extremely challenging task due to three conundrums: increasingly large model complexity, non-smooth representation shift during incremental learning and inconsistency between stage-wise sub-problem optimization and global inference. In this work, we propose the Discriminative Representation Learning (DRL) framework to specifically address these challenges. To conduct incremental learning effectively and yet efficiently, the DRL's network, called Incremental Parallel Adapter (IPA) network, is built upon a PTM and increasingly augments the model by learning a lightweight adapter with a small amount of parameter learning overhead in each incremental stage. The adapter is responsible for adapting the model to new classes, it can inherit and propagate the representation capability from the current model through parallel connection between them by a transfer gate. As a result, this design guarantees a smooth representation shift between different incremental stages. Furthermore, to alleviate inconsistency and enable comparable feature representations across incremental stages, we design the Decoupled Anchor Supervision (DAS). It decouples constraints of positive and negative samples by respectively comparing them with the virtual anchor. This decoupling promotes discriminative representation learning and aligns the feature spaces learned at different stages, thereby narrowing the gap between stage-wise local optimization over a subset of data and global inference across all classes. Extensive experiments on six benchmarks reveal that our DRL consistently outperforms other state-of-the-art methods throughout the entire CIL period while maintaining high efficiency in both training and inference phases.

Figures (7)

• Figure 1: Performance comparison of different class-incremental learning methods on ImageNet-A with B0 Inc20 setting, evaluated in terms of classification accuracy versus inference complexity measured by model size. The circle sizes represent the model size during inference phase, demonstrating the efficiency advantages of our DRL framework.
• Figure 2: Comparison of different CIL methods. We select three representative methods: DER, FOSTER, and EASE. '1B' denotes the total parameter of a model (e.g., ViT-B/16). a) DER and b) FOSTER face the increasingly large model complexity during the training. c) EASE faces an increasing computing overhead during inference, as well as non-smooth representation shift due to the independent training at different incremental stages. d) Our network mitigates these problems through the IPA module, which consists of a lightweight adapter with minimal parameter learning overhead at each incremental stage and a transfer gate to ensure a smooth representation shift.
• Figure 3: Detailed architecture of the Incremental Parallel Adapter (IPA) network. This design incorporates a lightweight adapter structure, a reusable attention mechanism, and a transfer gate to enable seamless knowledge propagation across incremental learning stages.
• Figure 4: Comparisons of decision boundaries. (a) Standard softmax exhibits inconsistent separation boundaries across incremental stages; (b) Our proposed DAS establishes consistent decision boundaries through a fixed virtual anchor, enabling stable representation learning across all incremental process.
• Figure 5: Incremental learning performance progression across multiple datasets using ViT-B/16-IN1K weights. The percentage values indicate DRL's relative improvement over the second-best method at the final task, demonstrating consistent superiority throughout the learning trajectory.