DiffClass: Diffusion-Based Class Incremental Learning

TL;DR

This work tackles exemplar-free Class Incremental Learning by addressing domain gaps between real and synthesized past data. It introduces a diffusion-based framework that integrates multi-distribution matching (MDM) with LoRA fine-tuning, selective synthetic image augmentation (SSIA), and multi-domain adaptation (MDA) to unify data distributions and balance stability with plasticity. Extensive experiments on CIFAR100 and ImageNet100 demonstrate state-of-the-art performance across multiple task splits, with notable gains as the number of tasks increases. The proposed approach provides a practical pathway for robust exemplar-free CIL in real-world settings by explicitly mitigating domain bias and enabling effective cross-domain knowledge transfer.

Abstract

Class Incremental Learning (CIL) is challenging due to catastrophic forgetting. On top of that, Exemplar-free Class Incremental Learning is even more challenging due to forbidden access to previous task data. Recent exemplar-free CIL methods attempt to mitigate catastrophic forgetting by synthesizing previous task data. However, they fail to overcome the catastrophic forgetting due to the inability to deal with the significant domain gap between real and synthetic data. To overcome these issues, we propose a novel exemplar-free CIL method. Our method adopts multi-distribution matching (MDM) diffusion models to unify quality and bridge domain gaps among all domains of training data. Moreover, our approach integrates selective synthetic image augmentation (SSIA) to expand the distribution of the training data, thereby improving the model's plasticity and reinforcing the performance of our method's ultimate component, multi-domain adaptation (MDA). With the proposed integrations, our method then reformulates exemplar-free CIL into a multi-domain adaptation problem to implicitly address the domain gap problem to enhance model stability during incremental training. Extensive experiments on benchmark class incremental datasets and settings demonstrate that our method excels previous exemplar-free CIL methods and achieves state-of-the-art performance.

Figures (6)

• Figure 1: Domain Gaps in Exemplar-Free CIL. The distribution of real classes is closer to each other while domain gaps exist between real class 0 and synthetic class 0.
• Figure 2: t-SNE Visualization of Test Data's Feature Embedding. Most of the previous task test data in incremental task 3 are misclassified as one of the task 3 classes.
• Figure 3: Model Framework Overview learning on currect task $\mathcal{T}_{i+1}$. previous MDM diffusion models $J_{0:i}$ are used to generated Synthetic Data of previous tasks $\mathcal{D}_{0:i}^{\text{syn}}$. MDM diffusion model of current task is then finetuned using MDM technique using Real current task Data $\mathcal{D}_{i}^{\text{real}}$ and randomly sampled small batch of $\mathcal{D}_{0:i}^{\text{syn}}$. $J_{0:i}$ is subsequently used to obtain $\mathcal{D}_{i}^{\text{aug}}$ by SSIA. The model trains with MDA on the combined dataset.
• Figure 4: Classification Accuracy of Each Incremental Task on CIFAR100. Our method greatly outperforms all data-free CIL baselines in all incremental settings.
• Figure 5: Incremental Accuracy on ImageNet100. Our method greatly outperforms all baseline methods in all incremental settings. Our method achieves more significant improvements in more incremental task settings (e.g. increase $N$ from 5 to 10 or to 20)