Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

No Forgetting Learning: Memory-free Continual Learning

Mohammad Ali Vahedifar, Qi Zhang

TL;DR

The paper tackles catastrophic forgetting in continual learning when past data are not stored. It introduces No Forgetting Learning (NFL), a memory-free distillation-based framework, and NFL+ with an auto-encoder to preserve features and address bias. NFL achieves competitive performance on Task-IL and Class-IL benchmarks while using roughly 14.75× less memory than memory-based methods, and it introduces the Plasticity-Stability Ratio to better quantify trade-offs. These results advocate memory-free CL as a scalable, privacy-friendly alternative and highlight the need for fair benchmarking that accounts for memory and computation.

Abstract

Continual Learning (CL) remains a central challenge in deep learning, where models must sequentially acquire new knowledge while mitigating Catastrophic Forgetting (CF) of prior tasks. Existing approaches often struggle with efficiency and scalability, requiring extensive memory or model buffers. This work introduces ``No Forgetting Learning" (NFL), a memory-free CL framework that leverages knowledge distillation to maintain stability while preserving plasticity. Memory-free means the NFL does not rely on any memory buffer. Through extensive evaluations of three benchmark datasets, we demonstrate that NFL achieves competitive performance while utilizing approximately 14.75 times less memory than state-of-the-art methods. Furthermore, we introduce a new metric to better assess CL's plasticity-stability trade-off.

No Forgetting Learning: Memory-free Continual Learning

TL;DR

The paper tackles catastrophic forgetting in continual learning when past data are not stored. It introduces No Forgetting Learning (NFL), a memory-free distillation-based framework, and NFL+ with an auto-encoder to preserve features and address bias. NFL achieves competitive performance on Task-IL and Class-IL benchmarks while using roughly 14.75× less memory than memory-based methods, and it introduces the Plasticity-Stability Ratio to better quantify trade-offs. These results advocate memory-free CL as a scalable, privacy-friendly alternative and highlight the need for fair benchmarking that accounts for memory and computation.

Abstract

Continual Learning (CL) remains a central challenge in deep learning, where models must sequentially acquire new knowledge while mitigating Catastrophic Forgetting (CF) of prior tasks. Existing approaches often struggle with efficiency and scalability, requiring extensive memory or model buffers. This work introduces ``No Forgetting Learning" (NFL), a memory-free CL framework that leverages knowledge distillation to maintain stability while preserving plasticity. Memory-free means the NFL does not rely on any memory buffer. Through extensive evaluations of three benchmark datasets, we demonstrate that NFL achieves competitive performance while utilizing approximately 14.75 times less memory than state-of-the-art methods. Furthermore, we introduce a new metric to better assess CL's plasticity-stability trade-off.

Paper Structure

This paper contains 13 sections, 33 equations, 7 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. No Forgetting Learning
  4. NFL
  5. NFL+
  6. Experiements
  7. Experiement Setup and Results
  8. Discussions
  9. Comparison on Small-Scale Datasets
  10. Comparison on large-Scale Datasets
  11. Conclusion
  12. Evaluation Protocol
  13. Additional Results

Figures (7)

  • Figure 1: A Conceptual Illustration of NFL.
  • Figure 2: ACC comparison for Class-IL using CIFAR-100 (10 incremental classes per task).
  • Figure 3: ACC comparison for Class-IL using TinyImageNet (20 incremental classes per task).
  • Figure 4: ACC comparison for Class-IL using ImageNet-1000 (100 incremental classes per task).
  • Figure 5: Memory size for ImageNet-1000 for different comparison methods. Blue bars denote the memory budget for exemplars, and orange bars denote the memory budget for keeping the model.
  • ...and 2 more figures