Recall-Oriented Continual Learning with Generative Adversarial Meta-Model

TL;DR

The paper tackles the stability-plasticity trade-off in continual learning by proposing a recall-oriented framework with a two-level memory architecture: a fast, task-specific working-memory inference network for plasticity and GAMM, a generative long-term memory that recalls past task parameters. It analyzes representation complexity and argues that parameter-level representations are more compact than raw data or feature vectors, motivating parameter-level generative replay. GAMM is trained as a conditional GAN that generatively replays parameter chunks from past tasks to consolidate knowledge while preserving past performance, and it can recall task-specific models at inference. Empirically, GAMM achieves state-of-the-art ACC with near-zero BWT in task-aware settings and strong LA in task-agnostic settings, with favorable memory usage compared to replay-based methods. This approach offers a scalable, memory-efficient path to robust continual learning by decoupling knowledge acquisition from recall and focusing on parameter-level representations.

Abstract

The stability-plasticity dilemma is a major challenge in continual learning, as it involves balancing the conflicting objectives of maintaining performance on previous tasks while learning new tasks. In this paper, we propose the recall-oriented continual learning framework to address this challenge. Inspired by the human brain's ability to separate the mechanisms responsible for stability and plasticity, our framework consists of a two-level architecture where an inference network effectively acquires new knowledge and a generative network recalls past knowledge when necessary. In particular, to maximize the stability of past knowledge, we investigate the complexity of knowledge depending on different representations, and thereby introducing generative adversarial meta-model (GAMM) that incrementally learns task-specific parameters instead of input data samples of the task. Through our experiments, we show that our framework not only effectively learns new knowledge without any disruption but also achieves high stability of previous knowledge in both task-aware and task-agnostic learning scenarios. Our code is available at: https://github.com/bigdata-inha/recall-oriented-cl-framework.

Figures (7)

• Figure 1: Trade-off between stability and plasticity in various CL baseline approaches on Split CIFAR-100.
• Figure 2: Separation of working memory and long-term memory in our framework.
• Figure 3: Separability and volume of input images, feature vectors, and parameter chunks.
• Figure 4: Illustration of three major steps in our recall-oriented continual leaning framework.
• Figure A1: T-SNE visualizations of input images and their corresponding parameter chunks, where each point, indicating either an image or a chunk, has a different color for a class or a chunk-id.