GradMix: Gradient-based Selective Mixup for Robust Data Augmentation in Class-Incremental Learning

TL;DR

GradMix addresses catastrophic forgetting in class-incremental learning by replacing random mixup with a gradient-based selective mixup that uses class-level gradient comparisons to avoid detrimental pairings. The method combines last-layer gradient approximations with a per-epoch class-pair optimization to maximize alignment with buffer-task gradients, thereby reducing forgetting while preserving augmentation benefits. Theoretical analysis and extensive experiments on MNIST, FMNIST, CIFAR-10/100, Tiny ImageNet, and ImageNet-1K show GradMix consistently outperforms standard mixup, imbalance-aware mixup, and policy-based augmentations, and remains effective with advanced experience-replay methods and different backbones. The approach offers a practical, scalable augmentation strategy for robust continual learning with limited buffer data and has potential for integration with transformer-based continual learning models.

Abstract

In the context of continual learning, acquiring new knowledge while maintaining previous knowledge presents a significant challenge. Existing methods often use experience replay techniques that store a small portion of previous task data for training. In experience replay approaches, data augmentation has emerged as a promising strategy to further improve the model performance by mixing limited previous task data with sufficient current task data. However, we theoretically and empirically analyze that training with mixed samples from random sample pairs may harm the knowledge of previous tasks and cause greater catastrophic forgetting. We then propose GradMix, a robust data augmentation method specifically designed for mitigating catastrophic forgetting in class-incremental learning. GradMix performs gradient-based selective mixup using a class-based criterion that mixes only samples from helpful class pairs and not from detrimental class pairs for reducing catastrophic forgetting. Our experiments on various real datasets show that GradMix outperforms data augmentation baselines in accuracy by minimizing the forgetting of previous knowledge.

Figures (57)

• Figure 1: Overview of GradMix. The gradients of the original samples and the mixed samples are denoted by $g$ and $\tilde{g}$, respectively. Given the average gradient of buffer data (denoted as $\overline{g}_{buf}$), GradMix performs gradient-based selective mixup to mitigate catastrophic forgetting by mixing only samples from helpful class pairs (e.g., 'Dog'--'Deer', where $\theta \leq 90^{\circ}$) and not from detrimental class pairs (e.g., 'Dog'--'Automobile', where $\theta > 90^{\circ}$).
• Figure 3: Average accuracy results on the CIFAR-10, CIFAR-100, and Tiny ImageNet datasets. We use MIR, GSS, DER, FOSTER, and MEMO as basis methods of class-incremental learning and integrate GradMix.
• Figure 4: Selective mixup results of GradMix on the last task for the MNIST, FMNIST, CIFAR-10, and CIFAR-100 datasets.
• Figure 11: Average cosine distances between gradients of data within the same class (intra-class) and from different classes (inter-class) on the MNIST, FMNIST, CIFAR-10, CIFAR-100, and Tiny ImageNet datasets.
• Figure : (a) Task-wise and class-wise accuracy (MNIST).

Theorems & Definitions (9)

• Theorem 1
• Lemma 1
• Theorem 2
• Theorem
• proof
• Lemma
• proof
• Theorem
• proof