REAL: Representation Enhanced Analytic Learning for Exemplar-free Class-incremental Learning

TL;DR

REAL addresses exemplar-free class-incremental learning by strengthening backbone representations and maximizing knowledge use from multiple backbone layers. It introduces dual-stream base pretraining to capture general and label-guided knowledge, followed by representation-enhancing distillation and a feature fusion buffer that yields informative multi-layer features for an analytic classifier. The approach preserves the ACL weight-invariance property and demonstrates state-of-the-art performance across CIFAR-100, ImageNet-100, and ImageNet-1k while remaining compatible with existing ACL methods. Overall, REAL improves stability and plasticity in EFCIL, reducing forgetting and benefiting from transferable features across phases.

Abstract

Exemplar-free class-incremental learning (EFCIL) aims to mitigate catastrophic forgetting in class-incremental learning (CIL) without available historical training samples as exemplars. Compared with its exemplar-based CIL counterpart that stores exemplars, EFCIL suffers more from forgetting issues. Recently, a new EFCIL branch named Analytic Continual Learning (ACL) introduces a gradient-free paradigm via Recursive Least-Square, achieving a forgetting-resistant classifier training with a frozen backbone during CIL. However, existing ACL suffers from ineffective representations and insufficient utilization of backbone knowledge. In this paper, we propose a representation-enhanced analytic learning (REAL) to address these problems. To enhance the representation, REAL constructs a dual-stream base pretraining followed by representation enhancing distillation process. The dual-stream base pretraining combines self-supervised contrastive learning for general features and supervised learning for class-specific knowledge, followed by the representation enhancing distillation to merge both streams, enhancing representations for subsequent CIL paradigm. To utilize more knowledge from the backbone, REAL presents a feature fusion buffer to multi-layer backbone features, providing informative features for the subsequent classifier training. Our method can be incorporated into existing ACL techniques and provides more competitive performance. Empirical results demonstrate that, REAL achieves state-of-the-art performance on CIFAR-100, ImageNet-100 and ImageNet-1k benchmarks, outperforming exemplar-free methods and rivaling exemplar-based approaches.

Figures (7)

• Figure 1: The difference between EBCIL and EFCIL. At each phase, EBCIL utilizes the stored exemplars selected from previous categories to train the model (as shown in (a)), while EFCIL does not keep exemplars and uses the categories within the training phase (i.e., in (b)).
• Figure 2: REAL focuses on enhancing the representation and backbone knowledge utilization during training and comprises three parts: (a) Dual-stream base pretraining (DS-BPT), (b) representation enhancing distillation (RED), and (c) analytic learning with feature fusion. Base knowledge is obtained by pretraining the CNN backbones with two streams. One stream acquires general base knowledge (GBK) via self-supervised contrastive learning (SSCL). The other learns supervised feature distribution (SFD) via supervised learning (i.e., in (a)). During RED, the backbone with SFD is frozen and transfers knowledge to the model with GBK via knowledge distillation (i.e., in (b)). Following base training, the enhanced backbone (i.e., backbone with GBK after RED) is embedded in the analytic learning agenda with a feature fusion buffer (i.e., in (c)).
• Figure 3: Phase-wise accuracy of compared methods on benchmark datasets with $K = 5, 10, 25$, and $50$ on CIFAR-100 and ImageNet-100. The results of ImageNet-1k with $K = 5, 10, 25$ are also included. The dash lines are the curves for exemplar-based methods. "CIFAR-100 (R32)" denotes the results on CIFAR-100 with backbone of ResNet-32, while "R18" indicates using the ResNet-18. Experiments without specific denotation use ResNet-18 as the backbone.
• Figure 4: Analysis on stability and ;plasticity by ablating RED and FFB.
• Figure 5: T-SNE plot of feature vectors extracted on ImageNet-100. The features extracted with REAL within a class are more compact compared with those in the baseline.