Partitioned Memory Storage Inspired Few-Shot Class-Incremental learning

TL;DR

This work tackles FSCIL by introducing a cortex-inspired partitioned-memory approach in which each incremental session is learned by a separate, session-specific model to prevent catastrophic forgetting. It combines a forward-compatible framework with virtual prototypes, a decoupled representation learning strategy that isolates parameters per session, and an uncertainty-based test-time model selection to map samples to the appropriate model. The method achieves state-of-the-art results on CIFAR-100 and mini-ImageNet, demonstrating strong robustness across sessions and validating the effectiveness of memory partitioning and UQ-driven selection. This approach provides a practical, biologically inspired perspective on FSCIL with meaningful implications for continual learning systems in dynamic environments.

Abstract

Current mainstream deep learning techniques exhibit an over-reliance on extensive training data and a lack of adaptability to the dynamic world, marking a considerable disparity from human intelligence. To bridge this gap, Few-Shot Class-Incremental Learning (FSCIL) has emerged, focusing on continuous learning of new categories with limited samples without forgetting old knowledge. Existing FSCIL studies typically use a single model to learn knowledge across all sessions, inevitably leading to the stability-plasticity dilemma. Unlike machines, humans store varied knowledge in different cerebral cortices. Inspired by this characteristic, our paper aims to develop a method that learns independent models for each session. It can inherently prevent catastrophic forgetting. During the testing stage, our method integrates Uncertainty Quantification (UQ) for model deployment. Our method provides a fresh viewpoint for FSCIL and demonstrates the state-of-the-art performance on CIFAR-100 and mini-ImageNet datasets.

Figures (5)

• Figure 1: Comparing ordinary incremental learning method to parameter-isolation method. $\theta^t$ represents the model parameter on session $t$. (a) Model parameters changes trigger catastrophic forgetting (b) The model for each session learns the data independently and save the parameters separately.
• Figure 2: The overview of our proposed humankind memory-inspired FSCIL approach. (a) The CutMix data augmentation method enhances the feature extractor's performance by generating virtual prototypes. For session $t$, freeze the bulk $\mathbf{F}_\phi$ of the feature extractor, fine-tune the tail $\mathbf{F}_e^t$ and the classifier $\mathbf{W}^t$. (b) In the testing stage, the samples $\mathbf{x}$ are fed to a trained series of models outputting categories probability $\mathbf{R}^t(\mathbf{x})$ and uncertainty $H^t(\mathbf{x})$. The reliable result is then selected based on the value of $H^t(\mathbf{x})$.
• Figure 3: Virtual prototypes in embedding space. (a) Samples determine their category based on the closest prototype to them. Samples are more likely to be misclassified when the prototypes are too close to each other. (b) Inserting the virtual prototypes increases the separation between real prototypes and the embedding space's sparsity. (c) Procedure for virtual sample generation.
• Figure 4: Decoupled training strategy in this paper. $\mathbf{x}$ represents the inputs. During incremental training, the majority of the feature extractor $\mathbf{F}_\phi$ is frozen to suppress catastrophic forgetting, while the tails $\mathbf{F}^t_e$ and classifiers undergo fine-tuning.
• Figure 5: Average accuracy in each session of two benchmark datasets: CIFAR-100, mini-ImageNet.