Remembering Transformer for Continual Learning

TL;DR

The paper tackles catastrophic forgetting in continual learning by introducing Remembering Transformer, a framework that fuses a Vision Transformer with a sparse mixture-of-adapters and a generative, autoencoder-based novelty detector to route tasks to the most relevant adapters without requiring task identities. It further enhances parameter efficiency through adapter fusion via knowledge distillation when capacity is limited. Empirical results across class-incremental split tasks and permutation tasks demonstrate state-of-the-art accuracy and substantially reduced memory footprints, highlighting the approach’s scalability to limited-capacity settings. The work advances practical continual learning by combining CLS-inspired consolidation, modular routing, and memory-efficient adaptation, enabling robust performance with compact models.

Abstract

Neural networks encounter the challenge of Catastrophic Forgetting (CF) in continual learning, where new task learning interferes with previously learned knowledge. Existing data fine-tuning and regularization methods necessitate task identity information during inference and cannot eliminate interference among different tasks, while soft parameter sharing approaches encounter the problem of an increasing model parameter size. To tackle these challenges, we propose the Remembering Transformer, inspired by the brain's Complementary Learning Systems (CLS). Remembering Transformer employs a mixture-of-adapters architecture and a generative model-based novelty detection mechanism in a pretrained Transformer to alleviate CF. Remembering Transformer dynamically routes task data to the most relevant adapter with enhanced parameter efficiency based on knowledge distillation. We conducted extensive experiments, including ablation studies on the novelty detection mechanism and model capacity of the mixture-of-adapters, in a broad range of class-incremental split tasks and permutation tasks. Our approach demonstrated SOTA performance surpassing the second-best method by 15.90% in the split tasks, reducing the memory footprint from 11.18M to 0.22M in the five splits CIFAR10 task.

Figures (6)

• Figure 1: Remembering Transformer leverages the mixture-of-adapters that are sparsely activated with a novelty detection mechanism.
• Figure 2: Adapter fusion based on knowledge distillation with a limited capacity $E$. We update the $E+1$-th adapter $\{B^l_{E+1}, A^l_{E+1}\}_{l=1}^L$ using new task data and the soft probability output of the old task replay.
• Figure 3: Test accuracy curves in the CIFAR100 tasks.
• Figure 4: Memory footprint of the comparison models.
• Figure 5: Test accuracy curves of the Remembering Transformer compared to the conventional methods for the permuted-MNIST task.