CD^2: Constrained Dataset Distillation for Few-Shot Class-Incremental Learning

TL;DR

This work tackles FSCIL by addressing catastrophic forgetting through CD^2, a framework that combines a Dataset Distillation Module to synthesize highly informative memory samples and a Distillation Constraint Module to stabilize knowledge transfer across sessions. The DDM reduces reliance on real data while preserving critical class-related information, and the DCM enforces feature and structural consistency to mitigate covariate shift. Together, these components yield state-of-the-art performance on CIFAR100, mini-ImageNet, and CUB200, with robust retention of old knowledge as new classes are introduced. The approach offers a practical memory-efficient solution for continual learning settings where labeled data in incremental sessions are scarce and privacy concerns exist.

Abstract

Few-shot class-incremental learning (FSCIL) receives significant attention from the public to perform classification continuously with a few training samples, which suffers from the key catastrophic forgetting problem. Existing methods usually employ an external memory to store previous knowledge and treat it with incremental classes equally, which cannot properly preserve previous essential knowledge. To solve this problem and inspired by recent distillation works on knowledge transfer, we propose a framework termed \textbf{C}onstrained \textbf{D}ataset \textbf{D}istillation (\textbf{CD$^2$}) to facilitate FSCIL, which includes a dataset distillation module (\textbf{DDM}) and a distillation constraint module~(\textbf{DCM}). Specifically, the DDM synthesizes highly condensed samples guided by the classifier, forcing the model to learn compacted essential class-related clues from a few incremental samples. The DCM introduces a designed loss to constrain the previously learned class distribution, which can preserve distilled knowledge more sufficiently. Extensive experiments on three public datasets show the superiority of our method against other state-of-the-art competitors.

Figures (5)

• Figure 1: Red lines are the gap between actual extracted features and ideal features with all critical knowledge. Green arrows are the covariate shift. (a) Previous methods build a memory that dilutes critical knowledge, and treat the memory as equal to the real data, causing the covariate shift. (b) Using dataset distillation, the method can build a memory with more critical knowledge, but also faces the challenge of the covariate shift. (c) Our method builds and uses a memory incorporating distillation, which gains more critical knowledge and reduces the covariate shift.
• Figure 2: The downtrend of accuracy for all sessions on CIFAR100 as an example. "Replay-1", "Replay-2", "Replay-3", "Replay-4", and "Replay-5" denote using sample replay to build a memory with 1, 2, 3, 4, and 5 images per class when training a model, which trains the model by optimizing Eq. \ref{['eq:replay']}. And "Prototype" means using prototypical features as a memory as Eq. \ref{['eq:prototype']} when training a model.
• Figure 3: The framework CD$^2$ generates memory by a dataset distillation module (DDM) and finetunes the model with a distillation constraint module (DCM), containing a backbone $\phi_b$, a classifier $\phi_c$, and an extra memory $\mathcal{M}^{(t)}$. During model training, we train the model on $\mathcal{D}^{(0)}$ in the base session ($t=0$), then freeze the backbone and finetune the classifier with DCM on dataset $\mathcal{D}^{(t)}$ and the memory $\mathcal{M}^{(t)}$ in $t$-th incremental session ($t>0$). And after model training for each session ($t<T$), we synthetic samples as $\mathcal{D}^{(t)}_{S}$ by DDM and send them to the memory $\mathcal{M}^{(t+1)}$.
• Figure 4: Performance curves of our method compared to recent SOTA methods on mini-ImageNet and CUB200. Left: mini-ImageNet. Right: CUB200. "Average" denotes the average accuracy of all sessions. Please refer to the appendix for more details.
• Figure 5: The t-SNE visualization of representations, which uses the base session and an incremental session on CIFAR-100 as an example. We randomly select 50 examples over 10 base classes and 5 incremental classes to show the effect. And we visualize all synthetic samples from all selected classes. Symbols like '$\bullet$' and '$\blacktriangle$' represent samples of base classes and incremental classes. '$\bigstar$' represents synthetic samples. (a) is visual features of real samples, and (b) is visual features of both real and synthetic samples. The top row shows the visualization of the base session, and the bottom row shows the visualization of the incremental session.