Unveiling the Tapestry: the Interplay of Generalization and Forgetting in Continual Learning

TL;DR

The paper addresses the gap between generalization and continual learning by showing a bidirectional relationship between forgetting and out-of-distribution robustness. It introduces Shape-Texture Consistency Regularization (STCR), a simple, plug-and-play regularizer that learns shape- and texture-based representations per task and enforces logit-consistency between original and shape-texture-conflicted inputs using a KL-divergence objective $\mathcal{L}^{STCR}$. STCR can be integrated with both replay and replay-free continual learning methods and, through extensive experiments on ImageNet-100/1000 and CIFAR-100, demonstrates significant improvements in both $\mathcal{R}$-C and forgetting metrics, often surpassing baselines and SOTA methods that pair with existing generalization techniques. The findings highlight that better generalization reduces forgetting and that effective forgetting mitigation enhances generalization, offering a path toward more robust continual learners. Limitations include compute-intensive style transfer and the supervised setting; future work could explore faster style-transfer alternatives and extend STCR to semi-supervised or unsupervised continual learning regimes.

Abstract

In AI, generalization refers to a model's ability to perform well on out-of-distribution data related to the given task, beyond the data it was trained on. For an AI agent to excel, it must also possess the continual learning capability, whereby an agent incrementally learns to perform a sequence of tasks without forgetting the previously acquired knowledge to solve the old tasks. Intuitively, generalization within a task allows the model to learn underlying features that can readily be applied to novel tasks, facilitating quicker learning and enhanced performance in subsequent tasks within a continual learning framework. Conversely, continual learning methods often include mechanisms to mitigate catastrophic forgetting, ensuring that knowledge from earlier tasks is retained. This preservation of knowledge over tasks plays a role in enhancing generalization for the ongoing task at hand. Despite the intuitive appeal of the interplay of both abilities, existing literature on continual learning and generalization has proceeded separately. In the preliminary effort to promote studies that bridge both fields, we first present empirical evidence showing that each of these fields has a mutually positive effect on the other. Next, building upon this finding, we introduce a simple and effective technique known as Shape-Texture Consistency Regularization (STCR), which caters to continual learning. STCR learns both shape and texture representations for each task, consequently enhancing generalization and thereby mitigating forgetting. Remarkably, extensive experiments validate that our STCR, can be seamlessly integrated with existing continual learning methods, including replay-free approaches. Its performance surpasses these continual learning methods in isolation or when combined with established generalization techniques by a large margin.

Figures (7)

• Figure 1: Illustration of the interplay between out-of-distribution generalization and continual learning. Over successive tasks, the model incrementally learns to recognize new object classes (black arrows), such as strawberries in Task 1 followed by birds in Task 2, and so on. During this continual learning process, the model exhibits a progressive loss of previously acquired knowledge known as catastrophic forgetting (orange arrows). Generalization refers to the model capability for performing well on the out-of-distribution data related to the given task, beyond the data it has seen during training (blue arrows). The sketch of the real-world strawberry in Task 1 refers to one out-of-distribution test sample, which was unseen by the model during training. Other examples for out-of-distribution data also include birds corrupted by noise or style-transferred cats. Here, we are to investigate the interplay of generalization within a given task and the forgetting about the previous tasks.
• Figure 2: Empirical evidence about the interplay of generalization and forgetting in continual learning on ImageNet-100 when $T=6$. Subplot (a) illustrates the forgetting ($\mathcal{F}$) and generalization capability ($\mathcal{R}$-C) across a range of continual learning methods. For each continual learning algorithm, we report its $\mathcal{R}$-C and $\mathcal{F}$. See Sec. \ref{['metrics']} for the definitions of $\mathcal{R}$-C and $\mathcal{F}$. We also performed a linear fitting (dashed line) between $\mathcal{R}$-C and $\mathcal{F}$ based on all the sample points on the subplot using RANSACRegressor teoh2015random. Subplot (b) presents the impact of generalization on reduced forgetting. We computed the performance difference in $\mathcal{R}$-C and $\mathcal{F}$ between existing continual learning baselines with and without the B-Aug generalization baseline, and denote these differences as $\Delta \mathcal{F}$ and $\Delta \mathcal{R}$-C. Postive $\Delta \mathcal{F}$ and $\Delta \mathcal{R}$-C imply that the continual learning model has improved generalization capability and reduced forgetting after integrating the B-Aug generalization algorithm, compared to its counterpart without B-Aug. The dashed line indicates the result of linear fitting between $\Delta F$ and $\Delta \mathcal{R}$-C using RANSACRegressor teoh2015random. See Sec. \ref{['subsec:baselines']} for the introduction of the continual learning baselines. See Sec. \ref{['a-baselines']} for the introduction of the generalization baselines. See Sec. \ref{['metrics']} for the introduction of the evaluation metrics.
• Figure 3: Loss landscape analysis on existing continual learning methods with and without existing data augmentation techniques. The loss landscape is produced by evaluating the final models at task $\mathcal{T}_6$ with perturbed weights in random directions on the test sets of tasks $\mathcal{T}_1$ (darker line) and $\mathcal{T}_6$ (lighter line) of the ImageNet-100 dataset. The x-axis indicates the magnitudes of weight perturbations. The results of the existing continual learning baselines in colors with the existing B-Aug generalization method (solid line) and without B-Aug (dotted line) are presented. See Sec. \ref{['subsec:baselines']} for the introduction of the continual learning baselines. See Sec. \ref{['a-baselines']} for the introduction of the generalization baselines.
• Figure 4: Schematic of our shape-texture consistency regularization method (STCR). Given an image from the training set, we create its shape-texture conflict counterpart by performing style transfers on the training images of the current task. The style images come from the examples stored in the memory buffer. The original image and shape-texture conflict image are combined in the mini-batch for training. The feature encoder extracts both texture-biased and shape-biased representations from these mini-batches. Their output logits are then normalized for classification via cross-entropy losses. Moreover, these logits are also regularized to make consistent class distributions. During replays, the model also rehearses the old images using cross-entropy losses. By using the proposed STCR, the resulting model can effectively prevent forgetting, and meanwhile, generalize to corrupted data, and domain-shifted data during inference. See Sec. \ref{['sec:stcr']} for the detailed design of our STCR.
• Figure 5: Loss landscape analysis on existing continual learning methods with and without our STCR in the ImageNet-100 dataset. We perform the loss landscape analysis using the final model $\theta_6$ at task $\mathcal{T}_6$ with perturbed weights in random directions on the test sets of tasks $\mathcal{T}_1$ (left panel) and $\mathcal{T}_6$ (right panel) of the ImageNet-100 dataset. These final models are trained with competitive continual learning baselines with and without our STCR. See Sec. \ref{['subsec:sotabaselines']} for the introduction to continual learning baselines.