Reducing Class-wise Confusion for Incremental Learning with Disentangled Manifolds

TL;DR

The work tackles class-wise confusion in class incremental learning (CIL) by combining per-class auto-encoders to learn disentangled latent manifolds with a confusion-aware latent-space separation loss. It introduces lightweight class-specific auto-encoders that map features $h = \phi(x)$ into class subspaces via encoders $f_i$ and decoders $g_i$, producing reconstruction-based probabilities $p_i = \exp(-\alpha \epsilon_i)/\sum_j \exp(-\alpha \epsilon_j)$ where $\epsilon_i = \|g_i(f_i(h)) - h\|$. A contrastive mechanism with confusion scores $s_i$ yields a weighted loss $L_{CR}$ through $w_i = 1 + \exp(-\beta s_i)$, integrated as $L = L_{CE} + L_{KD} + \lambda L_{CR}$ to encourage intra-class cohesion and inter-class separation in the latent space. Empirical results on CIFAR100 and ImageNet100 show CREATE achieving state-of-the-art performance, with up to $5.41\%$ gains in average accuracy and strong parameter efficiency, indicating practical improvements in robustness to feature drift. Overall, the approach provides a viable path to robust CIL by learning disentangled manifolds and enforcing targeted latent-space separation.

Abstract

Class incremental learning (CIL) aims to enable models to continuously learn new classes without catastrophically forgetting old ones. A promising direction is to learn and use prototypes of classes during incremental updates. Despite simplicity and intuition, we find that such methods suffer from inadequate representation capability and unsatisfied feature overlap. These two factors cause class-wise confusion and limited performance. In this paper, we develop a Confusion-REduced AuTo-Encoder classifier (CREATE) for CIL. Specifically, our method employs a lightweight auto-encoder module to learn compact manifold for each class in the latent subspace, constraining samples to be well reconstructed only on the semantically correct auto-encoder. Thus, the representation stability and capability of class distributions are enhanced, alleviating the potential class-wise confusion problem. To further distinguish the overlapped features, we propose a confusion-aware latent space separation loss that ensures samples are closely distributed in their corresponding low-dimensional manifold while keeping away from the distributions of features from other classes. Our method demonstrates stronger representational capacity and discrimination ability by learning disentangled manifolds and reduces class confusion. Extensive experiments on multiple datasets and settings show that CREATE outperforms other state-of-the-art methods up to 5.41%.

Figures (5)

• Figure 1: T-SNE visualization of feature distributions under CIFAR100 Base10 Inc10. The points marked with crosses represent features of the initial phase, while the points marked with circles indicate features of the final phase. Samples within the red box represent two instances of class confusion. (a) Class distributions and prototypes in the initial phase. Using a prototype to represent a class is incomplete. (b) A drift in incremental learning leads to alterations in class distribution and feature overlap. (c) Our method exhibits the ability to disentangle manifolds and reduce confusion.
• Figure 2: Overview of the proposed confusion-reduced auto-encoder classifier (CREATE). The auto-encoder (AE) learns a subspace for each class and generates a latent class distribution. Preserving the trained old AEs facilitates the retention of knowledge from old classes, while making these AEs trainable ensures adaptability to updates in the feature extractor $\phi_t$. To further alleviate class confusion in class incremental learning, we employ a confusion-aware separation loss $L_{CR}$ to separate samples from other classes within each subspace.
• Figure 3: Visualization of feature distributions on WA and Ours under CIFAR100 Base50 Inc5 final phase. CREATE can well distinguish class distributions.
• Figure 4: Analysis of confusion degree and hyper-parameter sensitivity.
• Figure 5: Reconstruction errors of misclassified data in CIFAR100 Base50 Inc5 phase2.