Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Discrimination and Consistency for Exemplar-Free Class Incremental Learning

Tianqi Wang, Jingcai Guo, Depeng Li, Zhi Chen

TL;DR

This work tackles exemplar-free class incremental learning (EF-CIL) by arguing that preserving a discriminative and consistent feature space enables inter-task interaction without replay. It proposes DCNet, a multi-head architecture with Incremental Orthogonal Embedding (IOE) and Dynamic Aggregation Compensation (DAC), guided by Hard Attention Masks (HAT) to protect task-specific knowledge and maintain inter-task discrimination. Theoretical analysis shows that better inter-class separation and tighter intra-class aggregation improve OOD detection and facilitate task information interaction in EF-CIL, leading to a TIL+OOD-inspired framework tailored to privacy constraints. Empirical results across CIFAR-100, Tiny-ImageNet, and ImageNet-Subset demonstrate that DCNet achieves competitive or superior performance to state-of-the-art EF-CIL methods, with notable gains without replay buffers and robust interaction across incremental tasks via IOE and DAC (e.g., significant improvements on ImageNet-Subset). The approach offers a privacy-preserving, interaction-enabled pathway for EF-CIL with practical impact on real-world continual learning deployments.

Abstract

Exemplar-free class incremental learning (EF-CIL) is a nontrivial task that requires continuously enriching model capability with new classes while maintaining previously learned knowledge without storing and replaying any old class exemplars. An emerging theory-guided framework for CIL trains task-specific models for a shared network, shifting the pressure of forgetting to task-id prediction. In EF-CIL, task-id prediction is more challenging due to the lack of inter-task interaction (e.g., replays of exemplars). To address this issue, we conduct a theoretical analysis of the importance and feasibility of preserving a discriminative and consistent feature space, upon which we propose a novel method termed DCNet. Concretely, it progressively maps class representations into a hyperspherical space, in which different classes are orthogonally distributed to achieve ample inter-class separation. Meanwhile, it also introduces compensatory training to adaptively adjust supervision intensity, thereby aligning the degree of intra-class aggregation. Extensive experiments and theoretical analysis verified the superiority of the proposed DCNet.

On the Discrimination and Consistency for Exemplar-Free Class Incremental Learning

TL;DR

This work tackles exemplar-free class incremental learning (EF-CIL) by arguing that preserving a discriminative and consistent feature space enables inter-task interaction without replay. It proposes DCNet, a multi-head architecture with Incremental Orthogonal Embedding (IOE) and Dynamic Aggregation Compensation (DAC), guided by Hard Attention Masks (HAT) to protect task-specific knowledge and maintain inter-task discrimination. Theoretical analysis shows that better inter-class separation and tighter intra-class aggregation improve OOD detection and facilitate task information interaction in EF-CIL, leading to a TIL+OOD-inspired framework tailored to privacy constraints. Empirical results across CIFAR-100, Tiny-ImageNet, and ImageNet-Subset demonstrate that DCNet achieves competitive or superior performance to state-of-the-art EF-CIL methods, with notable gains without replay buffers and robust interaction across incremental tasks via IOE and DAC (e.g., significant improvements on ImageNet-Subset). The approach offers a privacy-preserving, interaction-enabled pathway for EF-CIL with practical impact on real-world continual learning deployments.

Abstract

Exemplar-free class incremental learning (EF-CIL) is a nontrivial task that requires continuously enriching model capability with new classes while maintaining previously learned knowledge without storing and replaying any old class exemplars. An emerging theory-guided framework for CIL trains task-specific models for a shared network, shifting the pressure of forgetting to task-id prediction. In EF-CIL, task-id prediction is more challenging due to the lack of inter-task interaction (e.g., replays of exemplars). To address this issue, we conduct a theoretical analysis of the importance and feasibility of preserving a discriminative and consistent feature space, upon which we propose a novel method termed DCNet. Concretely, it progressively maps class representations into a hyperspherical space, in which different classes are orthogonally distributed to achieve ample inter-class separation. Meanwhile, it also introduces compensatory training to adaptively adjust supervision intensity, thereby aligning the degree of intra-class aggregation. Extensive experiments and theoretical analysis verified the superiority of the proposed DCNet.
Paper Structure (23 sections, 6 theorems, 48 equations, 4 figures, 3 tables, 1 algorithm)

This paper contains 23 sections, 6 theorems, 48 equations, 4 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Theoretical Analysis
  4. Class-Incremental Learning Setup
  5. OOD Detection Capabilities
  6. Interpretation.
  7. Task Information Interaction
  8. Methodology
  9. Overview
  10. Incremental Orthogonal Embedding (IOE)
  11. Dynamic Aggregation Compensation (DAC)
  12. Experiment
  13. Experiment Setting
  14. Main Comparison Results
  15. Algorithm Analysis
  16. ...and 8 more sections

Key Result

Lemma 1

Let $\alpha_{i,t} := \tfrac{1}{2}\, d_M\!\bigl(\mu_{\mathrm{in},i}, \mu_{\mathrm{out},t}\bigr),\; i = 1,\ldots, k,\; t = 1,\ldots, T$, then we have the following estimate: Details of the proof are provided in Appendix proof2.

Figures (4)

  • Figure 1: Overview of DCNet. Upon the arrival of task t, DCNet optimizes the learnable part of the backbone and creates a new OOD classifier. Through IOE, DCNet incrementally embeds new features in directions that remain orthogonal to previous categories. Subsequently, the DAC module dynamically compensates for any insufficient aggregation by referencing the degree of aggregation from the previous tasks.
  • Figure 2: t-SNE visualization of the embedding space, where a color represents a category. (a) (b) illustrate the embedding results for HAT+CSI; and (c) (d) present results for our method.
  • Figure 3: Changes in the degree of aggregation $\omega^{(t)}$.
  • Figure 4: Average Mahalanobis distance between classes.

Theorems & Definitions (9)

  • Lemma 1
  • Theorem 1
  • Lemma 2
  • proof
  • Lemma 3
  • Lemma 4
  • Lemma 5
  • proof
  • proof