Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DS-AL: A Dual-Stream Analytic Learning for Exemplar-Free Class-Incremental Learning

Huiping Zhuang, Run He, Kai Tong, Ziqian Zeng, Cen Chen, Zhiping Lin

TL;DR

This work tackles exemplar-free class-incremental learning by introducing DS-AL, a dual-stream framework that blends a main stream based on a Concatenated Recursive Least Squares (C-RLS) formulation with a compensation stream governed by a Dual-Activation Compensation (DAC) module. The main stream recasts CIL as a phase-wise recursive linear update, yielding an equivalence to joint learning and enabling phase-invariant performance without replay data. The compensation stream uses a residue-based target and a different activation to project into the main mapping's null space, with a PLC step to curb overcompensation; together these components boost fitting and plasticity. Empirical results on CIFAR-100, ImageNet-100, and ImageNet-Full show DS-AL achieving competitive or superior performance to exemplar-free baselines and approaching or surpassing many replay-based methods, even in large-scale, hundreds-of-phases settings. The approach offers a scalable, privacy-preserving alternative for continual learning with closed-form updates and strong empirical robustness.

Abstract

Class-incremental learning (CIL) under an exemplar-free constraint has presented a significant challenge. Existing methods adhering to this constraint are prone to catastrophic forgetting, far more so than replay-based techniques that retain access to past samples. In this paper, to solve the exemplar-free CIL problem, we propose a Dual-Stream Analytic Learning (DS-AL) approach. The DS-AL contains a main stream offering an analytical (i.e., closed-form) linear solution, and a compensation stream improving the inherent under-fitting limitation due to adopting linear mapping. The main stream redefines the CIL problem into a Concatenated Recursive Least Squares (C-RLS) task, allowing an equivalence between the CIL and its joint-learning counterpart. The compensation stream is governed by a Dual-Activation Compensation (DAC) module. This module re-activates the embedding with a different activation function from the main stream one, and seeks fitting compensation by projecting the embedding to the null space of the main stream's linear mapping. Empirical results demonstrate that the DS-AL, despite being an exemplar-free technique, delivers performance comparable with or better than that of replay-based methods across various datasets, including CIFAR-100, ImageNet-100 and ImageNet-Full. Additionally, the C-RLS' equivalent property allows the DS-AL to execute CIL in a phase-invariant manner. This is evidenced by a never-before-seen 500-phase CIL ImageNet task, which performs on a level identical to a 5-phase one. Our codes are available at https://github.com/ZHUANGHP/Analytic-continual-learning.

DS-AL: A Dual-Stream Analytic Learning for Exemplar-Free Class-Incremental Learning

TL;DR

This work tackles exemplar-free class-incremental learning by introducing DS-AL, a dual-stream framework that blends a main stream based on a Concatenated Recursive Least Squares (C-RLS) formulation with a compensation stream governed by a Dual-Activation Compensation (DAC) module. The main stream recasts CIL as a phase-wise recursive linear update, yielding an equivalence to joint learning and enabling phase-invariant performance without replay data. The compensation stream uses a residue-based target and a different activation to project into the main mapping's null space, with a PLC step to curb overcompensation; together these components boost fitting and plasticity. Empirical results on CIFAR-100, ImageNet-100, and ImageNet-Full show DS-AL achieving competitive or superior performance to exemplar-free baselines and approaching or surpassing many replay-based methods, even in large-scale, hundreds-of-phases settings. The approach offers a scalable, privacy-preserving alternative for continual learning with closed-form updates and strong empirical robustness.

Abstract

Class-incremental learning (CIL) under an exemplar-free constraint has presented a significant challenge. Existing methods adhering to this constraint are prone to catastrophic forgetting, far more so than replay-based techniques that retain access to past samples. In this paper, to solve the exemplar-free CIL problem, we propose a Dual-Stream Analytic Learning (DS-AL) approach. The DS-AL contains a main stream offering an analytical (i.e., closed-form) linear solution, and a compensation stream improving the inherent under-fitting limitation due to adopting linear mapping. The main stream redefines the CIL problem into a Concatenated Recursive Least Squares (C-RLS) task, allowing an equivalence between the CIL and its joint-learning counterpart. The compensation stream is governed by a Dual-Activation Compensation (DAC) module. This module re-activates the embedding with a different activation function from the main stream one, and seeks fitting compensation by projecting the embedding to the null space of the main stream's linear mapping. Empirical results demonstrate that the DS-AL, despite being an exemplar-free technique, delivers performance comparable with or better than that of replay-based methods across various datasets, including CIFAR-100, ImageNet-100 and ImageNet-Full. Additionally, the C-RLS' equivalent property allows the DS-AL to execute CIL in a phase-invariant manner. This is evidenced by a never-before-seen 500-phase CIL ImageNet task, which performs on a level identical to a 5-phase one. Our codes are available at https://github.com/ZHUANGHP/Analytic-continual-learning.
Paper Structure (23 sections, 1 theorem, 19 equations, 5 figures, 2 tables)

This paper contains 23 sections, 1 theorem, 19 equations, 5 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Replay-based CIL
  4. Exemplar-free CIL
  5. Regularization-based CIL
  6. Prototype-based CIL
  7. Analytic Learning based CIL
  8. The Proposed Method
  9. BP-based Training
  10. The Main Stream of DS-AL
  11. AL-based Re-training.
  12. The AL-based CIL.
  13. The Compensation Stream of DS-AL
  14. Experiments
  15. Experimental Setup
  16. ...and 8 more sections

Key Result

Theorem 1

Let $\bm{\hat{W}}_{\text{M}}^{(k)}$ be the optimal estimation of $\bm{ W}_{\text{M}}^{(k)}$ using eq_w_k-1 with all the training data from phase $0$ to $k$. Let $\bm{\hat{W}}_{\text{M}}^{(k-1)\prime} = [\bm{\hat{W}}_{\text{M}}^{(k-1)} \ \ \ \bm{0}]$, and $\bm{\hat{W}}_{\text{M}}^{(k)}$ can be equiva where

Figures (5)

  • Figure 1: The DS-AL consists of (a) BP-based training on the base dataset, followed by (b)-(d) AL-based training steps. Each step includes a main stream (green block) with a C-RLS module and a compensation stream (orange block) using the mapping residue from main stream as the label. (b) DS-AL initializes CIL by replacing and re-training the classification head with an AL-based one that includes a buffer layer and a linear classifier. (c)-(d) The CIL is then recursively conducted, incorporating a PLC module (defined in \ref{['eq_plc']}) to ensure the incremental constraint in the compensation stream.
  • Figure 2: The change of top-5 predictions of images w/ and w/o compensation ($\mathcal{C}=1.0$). The compensation improves the fitting as well as generalization abilities.
  • Figure 3: (a) Stability-plasticity changes via the DAC module. (b) Analysis of compensation ratio $\mathcal{C}$ on CIFAR-100.
  • Figure 4: Last-phase accuracy of DS-AL with different activation functions and compensation ratio.
  • Figure 5: The evolution of $\mathcal{\bar{A}}$ with the growing $K$.

Theorems & Definitions (2)

  • Theorem 1
  • proof