Task-Adaptive Saliency Guidance for Exemplar-free Class Incremental Learning

TL;DR

Exemplar-free Class Incremental Learning (EFCIL) struggles with forgetting as model attention drifts from task-relevant regions. The paper introduces Task-Adaptive Saliency Supervision (TASS), which combines boundary-guided saliency, task-agnostic low-level supervision, and saliency noise injection to regularize and stabilize attention across tasks. TASS can be integrated with existing EFCIL methods and achieves state-of-the-art results on CIFAR-100, Tiny-ImageNet, and ImageNet-Subset, with pronounced gains on longer task sequences. Ablation studies and visual analyses corroborate that the approach reduces saliency drift and yields more discriminative representations with robust saliency preservation.

Abstract

Exemplar-free Class Incremental Learning (EFCIL) aims to sequentially learn tasks with access only to data from the current one. EFCIL is of interest because it mitigates concerns about privacy and long-term storage of data, while at the same time alleviating the problem of catastrophic forgetting in incremental learning. In this work, we introduce task-adaptive saliency for EFCIL and propose a new framework, which we call Task-Adaptive Saliency Supervision (TASS), for mitigating the negative effects of saliency drift between different tasks. We first apply boundary-guided saliency to maintain task adaptivity and \textit{plasticity} on model attention. Besides, we introduce task-agnostic low-level signals as auxiliary supervision to increase the \textit{stability} of model attention. Finally, we introduce a module for injecting and recovering saliency noise to increase the robustness of saliency preservation. Our experiments demonstrate that our method can better preserve saliency maps across tasks and achieve state-of-the-art results on the CIFAR-100, Tiny-ImageNet, and ImageNet-Subset EFCIL benchmarks. Code is available at \url{https://github.com/scok30/tass}.

Figures (7)

• Figure 1: We propose the TASS method, which can be directly applied to many recent exemplar-free class incremental learning methods, resulting in a significant improvement in EFCIL classification accuracy and a reduction in catastrophic forgetting.
• Figure 2: Overall framework of Task-Adaptive Saliency Supervision (TASS). We apply a low-level model to generate saliency and boundary maps. The boundary map is dilated and downsampled to provide supervision at different stages of the encoder. A decoder is attached after the encoder for low-level distillation, which serves as stationary task-agnostic saliency guidance. To prevent saliency drift in later training phases, we introduce saliency noise into each encoder stage. The model is trained to denoise and reduce the saliency drift on current data in future phases. TASS can be integrated into an EFCIL approach to provide robust saliency guidance across incremental tasks.
• Figure 3: We dilate the boundary map and apply a binary cross entropy loss at three stages in the CNN backbone to prevent mid-level attention from drifting into boundary regions.
• Figure 4: Results on Tiny-ImageNet and ImageNet-Subset for different numbers of tasks. Our method outperforms others, especially on longer task sequences (i.e. more, but smaller, tasks).
• Figure 5: Visualization of the saliency (a) and boundary (b) maps from our student encoder-decoder network with original images from different tasks at different stages of incremental learning. Our method produces stable low-level results while reducing forgetting in classification. (c) The MAE loss between the student and teacher network across tasks.