Mimicking the Oracle: An Initial Phase Decorrelation Approach for Class Incremental Learning

TL;DR

This paper tackles Class Incremental Learning (CIL) by focusing on the initial phase, where representations strongly influence future forgetting. It introduces Class-wise Decorrelation (CwD), a simple regularizer that nudges per-class representations to scatter more uniformly, effectively mimicking an oracle model trained on all classes. The authors first show that directly mimicking the oracle at the initial phase can substantially improve CIL, then justify the decorrelation approach with eigenvalue analyses of class covariances and derive a practical CwD objective: $L_{\mathrm{CwD}}(\theta)=\frac{1}{C \cdot d^2} \sum_{c=1}^{C} \|K^{(c)}\|_F^2$ integrated as $\min_{\theta} L_{ce} + \eta L_{\mathrm{CwD}}$. Across CIFAR100, ImageNet100, and ImageNet, CwD yields consistent 1–3% gains over state-of-the-art baselines, especially in longer incremental sequences, validating its effectiveness as a plug-in regularizer with broad practical impact.

Abstract

Class Incremental Learning (CIL) aims at learning a multi-class classifier in a phase-by-phase manner, in which only data of a subset of the classes are provided at each phase. Previous works mainly focus on mitigating forgetting in phases after the initial one. However, we find that improving CIL at its initial phase is also a promising direction. Specifically, we experimentally show that directly encouraging CIL Learner at the initial phase to output similar representations as the model jointly trained on all classes can greatly boost the CIL performance. Motivated by this, we study the difference between a naïvely-trained initial-phase model and the oracle model. Specifically, since one major difference between these two models is the number of training classes, we investigate how such difference affects the model representations. We find that, with fewer training classes, the data representations of each class lie in a long and narrow region; with more training classes, the representations of each class scatter more uniformly. Inspired by this observation, we propose Class-wise Decorrelation (CwD) that effectively regularizes representations of each class to scatter more uniformly, thus mimicking the model jointly trained with all classes (i.e., the oracle model). Our CwD is simple to implement and easy to plug into existing methods. Extensive experiments on various benchmark datasets show that CwD consistently and significantly improves the performance of existing state-of-the-art methods by around 1\% to 3\%. Code will be released.

Figures (11)

• Figure 1: Visualization of representations (normalized to the unit sphere) in a two-phase CIL setting (learning 2 classes for each phase). (a) Naïve training at the initial phase (a.k.a., phase $0$). The data representations of each class lie in a long and narrow region. (b) Joint training on all 4 classes (oracle model). The data representations of each class scatter more uniformly. (c) Directly mimicking the oracle model at the initial phase, i.e., training the CIL learner with a regularization term that enforces the learner to output representation that is similar to the oracle model. This makes the representations of each class scatter more uniformly (like (b)). (d) Training at the initial phase with our CwD regularizer, which also yields uniformly-scattered representations (like (b) and (c)). Best viewed in color.
• Figure 2: The effectiveness of directly the mimicking the representations of the oracle model at the initial phase. (a) Initially trained on 50 classes, and then incremented with 10 classes per phase for 5 more phases. (b) Initially trained on 10 classes and then incremented with 10 classes per phase for 9 more phases. The regularization coefficient $\beta$ is defined in Eqn. \ref{['oracle_obj']}. We show the accuracy of each CIL phases. Results are averaged over 3 runs.
• Figure 3: Visualization on how $\alpha^{(c)}_{k}$ changes with increasing $k$ for models trained with different number of classes.$\alpha^{(c)}_{k}$, which measures the proportion of variance represented by the top $k$ eigenvalues, is defined in Eqn. \ref{['alpha_k']}. We plot curve of $\alpha^{(c)}_{k}$ for ResNet18 models trained with 10/25/50/100 ImageNet classes.
• Figure 4: Effects of class-wise decorrelation on representations of each class.$\alpha^{(c)}_{k}$, which measures the proportion of variance represented by the top $k$ eigenvalues, is defined in Eqn. \ref{['alpha_k']}. $\eta$ is the CwD coefficient used in Eqn. \ref{['overall_obj']}. We plot curve of $\alpha^{(c)}_{k}$ with or without CwD objective when training with 10/25/50 classes. We also plot the curve for the model trained on all 100 classes for comparison.
• Figure 5: Accuracy at each phase. With ImageNet100, learning 50 classes at initial phase and 5 classes per phase for the rest 50 classes. Shading areas denote standard deviation.

Theorems & Definitions (2)

• Proposition 1
• proof