GUIDE: Guidance-based Incremental Learning with Diffusion Models

TL;DR

GUIDE introduces classifier-guided diffusion-based rehearsal to mitigate forgetting in class-incremental learning. By steering the diffusion denoising process toward current-task classes, GUIDE produces rehearsal samples near decision boundaries, enhancing the retention of past knowledge while learning new tasks. Across CIFAR-10/100 and ImageNet-100 benchmarks, GUIDE surpasses prior generative replay methods in final accuracy and forgetting metrics, revealing a favorable stability-plasticity trade-off. The work also analyzes variants and runtime considerations, highlighting the practical value and current limitations of integrating classifier guidance with diffusion-based continual learning.

Abstract

We introduce GUIDE, a novel continual learning approach that directs diffusion models to rehearse samples at risk of being forgotten. Existing generative strategies combat catastrophic forgetting by randomly sampling rehearsal examples from a generative model. Such an approach contradicts buffer-based approaches where sampling strategy plays an important role. We propose to bridge this gap by incorporating classifier guidance into the diffusion process to produce rehearsal examples specifically targeting information forgotten by a continuously trained model. This approach enables the generation of samples from preceding task distributions, which are more likely to be misclassified in the context of recently encountered classes. Our experimental results show that GUIDE significantly reduces catastrophic forgetting, outperforming conventional random sampling approaches and surpassing recent state-of-the-art methods in continual learning with generative replay.

Figures (13)

• Figure 1: Rehearsal sampling in GUIDE. We guide the denoising process of a diffusion model trained on the previous task (blue) toward classes from the current task (orange). The replay samples, highlighted with blue borders, share features with the examples from the current task, which may be related to characteristics such as color or background (e.g., fishes on a snowy background when guided to snowmobile). Generative rehearsal on such samples positioned near the classifier's decision boundary successfully mitigates catastrophic forgetting.
• Figure 2: Samples from the unconditional diffusion model trained only on goldfish and tiger shark classes from the ImageNet100 dataset. In the upper row, we present the samples guided to the goldfish class, while in the bottom row, to the tiger shark class. At the same time, the classifier guides the denoising process toward the class depicted above each figure that was not included in the training set of the diffusion model. For reference, in the leftmost column, we present samples generated without guidance toward any unknown class, setting $s_2 = 0$. In every other column, we set both $s_1$ and $s_2$ to $10$. We obtain samples from the desired class with observable features of classes unknown to the diffusion model, such as the color, background, or shape.
• Figure 3: Accuracy on each task during each phase of class-incremental training on CIFAR-100 with 5 tasks - standard GR with diffusion (left), GFR (middle), and our method (right). We observe the stability-plasticity trade-off, where our method significantly reduces forgetting compared to the standard GR scenario at the cost of a slight decrease in the ability to learn new tasks.
• Figure 4: Visualization of the classifiers embedding space (umap) for training examples and generations sampled with standard generative replay method (left) and ours (right) at 75% of the training on the second task. We can observe how GUIDE sample generations are more similar to the training examples from new classes (e.g., airplanes similar to birds).
• Figure 5: Classifier scale impact on forgetting and ability to acquire new information. Up to $s=0.2$, when we increase scale, we reduce the forgetting but also observe a drop in the accuracy on the second task. When we use too large scale $s$, the quality of samples drops significantly, along with the accuracy on the previous task. We further present this effect in Appendix \ref{['app:rehearsal_scale']}.