Fixed Random Classifier Rearrangement for Continual Learning

TL;DR

This work addresses catastrophic forgetting in continual learning by highlighting the underexplored role of the classifier. It introduces Fixed Random Classifier Rearrangement (FRCR), a two-stage method that fixes a random classifier to bound the norm of the equivalent one-class classifier $\hat{\mathbf{w}}$ and then rearranges classifier entries to minimize cross-task alignment, thereby reducing latent drift. The authors provide a theoretical analysis showing that cross-task drift scales with $| (\hat{\mathbf{w}}^{A})^T \hat{\mathbf{w}}^{B} |$, and demonstrate through experiments on 5-Split MNIST/FashionMNIST/CIFAR10 that FRCR substantially mitigates forgetting with modest computational overhead. The work contributes a principled, data-free approach to continual learning that improves stability without replay or parameter growth, and offers insights into how classifier dynamics influence representation drift. Limitations include applicability mainly to binary tasks and shallower networks, suggesting future work to extend to multi-class settings and deeper models.

Abstract

With the explosive growth of data, continual learning capability is increasingly important for neural networks. Due to catastrophic forgetting, neural networks inevitably forget the knowledge of old tasks after learning new ones. In visual classification scenario, a common practice of alleviating the forgetting is to constrain the backbone. However, the impact of classifiers is underestimated. In this paper, we analyze the variation of model predictions in sequential binary classification tasks and find that the norm of the equivalent one-class classifiers significantly affects the forgetting level. Based on this conclusion, we propose a two-stage continual learning algorithm named Fixed Random Classifier Rearrangement (FRCR). In first stage, FRCR replaces the learnable classifiers with fixed random classifiers, constraining the norm of the equivalent one-class classifiers without affecting the performance of the network. In second stage, FRCR rearranges the entries of new classifiers to implicitly reduce the drift of old latent representations. The experimental results on multiple datasets show that FRCR significantly mitigates the model forgetting; subsequent experimental analyses further validate the effectiveness of the algorithm.

Figures (4)

• Figure 1: The accuracy and trajectories
• Figure 2: Classifier Entries Rearrangement Algorithm
• Figure 3: Clustering results of learnable classifiers versus fixed classifiers
• Figure 4: Correlation of latent representations ${\mathbf{h}}_1$ of category 2 images after all tasks finish.