Restoring Forgotten Knowledge in Non-Exemplar Class Incremental Learning through Test-Time Semantic Evolution

TL;DR

This paper tackles forgetting in Non-exemplar Class Incremental Learning (NECIL) by introducing RoSE, a test-time semantic evolution framework that leverages test data to correct semantic drift of old classes. RoSE trains an online, self-supervised auxiliary task to estimate class-prototype drift and replaces gradient-based updates with an analytical solution for the projector, enabling stable online adaptation. Empirical results on CIFAR100, TinyImageNet, and ImageNet100 show RoSE achieving state-of-the-art performance in both cold-start and warm-start NECIL settings, with significant gains over existing methods. The work demonstrates the practical value of exploiting test-time information to restore forgotten knowledge, suggesting a broader paradigm where testing stages contribute to continual learning effectiveness.

Abstract

Continual learning aims to accumulate knowledge over a data stream while mitigating catastrophic forgetting. In Non-exemplar Class Incremental Learning (NECIL), forgetting arises during incremental optimization because old classes are inaccessible, hindering the retention of prior knowledge. To solve this, previous methods struggle in achieving the stability-plasticity balance in the training stages. However, we note that the testing stage is rarely considered among them, but is promising to be a solution to forgetting. Therefore, we propose RoSE, which is a simple yet effective method that \textbf{R}est\textbf{o}res forgotten knowledge through test-time \textbf{S}emantic \textbf{E}volution. Specifically designed for minimizing forgetting, RoSE is a test-time semantic drift compensation framework that enables more accurate drift estimation in a self-supervised manner. Moreover, to avoid incomplete optimization during online testing, we derive an analytical solution as an alternative to gradient descent. We evaluate RoSE on CIFAR-100, TinyImageNet, and ImageNet100 datasets, under both cold-start and warm-start settings. Our method consistently outperforms most state-of-the-art (SOTA) methods across various scenarios, validating the potential and feasibility of test-time evolution in NECIL.

Figures (6)

• Figure 1: In the training stages of NECIL, the model only learns from the new task data distribution, which has a gap with the previous data distribution. Traditional NECIL testing cannot update the model with the test distribution to avoid forgetting the old data distribution. We propose to utilize test data during each incremental task's testing to restore knowledge forgotten in the training stages.
• Figure 2: Test process of RoSE. We maintain two feature queues, $Q_{old}$ and $Q_{new}$ during testing. A) At test start, we generate sufficient pseudo-features to initialize $Q_{old}$. These pseudo-features are then fed into the projector to initialize $Q_{new}$. B) During each incremental task's testing, the projector is updated by online test data to better evolve the features and prototypes. C) Through the updated projector, we estimate the positions of old prototypes in the new feature space for image classification.
• Figure 3: Relation between the accuracy and the number of test samples per class for baseline, gradient descent, gradient descent with a queue of the same capacity as RoSE, and RoSE.
• Figure 4: Accuracy of different methods on old and new classes. 'CS', 'WS' denotes 'cold-start' and 'warm-start'.
• Figure 5: Accuracy after each task in the cold-start 10 tasks settings.