Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prototypical Exemplar Condensation for Memory-efficient Online Continual Learning

Minh-Duong Nguyen, Thien-Thanh Dao, Le-Tuan Nguyen, Dung D. Le, Kok-Seng Wong

Abstract

Rehearsal-based continual learning (CL) mitigates catastrophic forgetting by maintaining a subset of samples from previous tasks for replay. Existing studies primarily focus on optimizing memory storage through coreset selection strategies. While these methods are effective, they typically require storing a substantial number of samples per class (SPC), often exceeding 20, to maintain satisfactory performance. In this work, we propose to further compress the memory footprint by synthesizing and storing prototypical exemplars, which can form representative prototypes when passed through the feature extractor. Owing to their representative nature, these exemplars enable the model to retain previous knowledge using only a small number of samples while preserving privacy. Moreover, we introduce a perturbation-based augmentation mechanism that generates synthetic variants of previous data during training, thereby enhancing CL performance. Extensive evaluations on widely used benchmark datasets and settings demonstrate that the proposed algorithm achieves superior performance compared to existing baselines, particularly in scenarios involving large-scale datasets and a high number of tasks.

Prototypical Exemplar Condensation for Memory-efficient Online Continual Learning

Abstract

Rehearsal-based continual learning (CL) mitigates catastrophic forgetting by maintaining a subset of samples from previous tasks for replay. Existing studies primarily focus on optimizing memory storage through coreset selection strategies. While these methods are effective, they typically require storing a substantial number of samples per class (SPC), often exceeding 20, to maintain satisfactory performance. In this work, we propose to further compress the memory footprint by synthesizing and storing prototypical exemplars, which can form representative prototypes when passed through the feature extractor. Owing to their representative nature, these exemplars enable the model to retain previous knowledge using only a small number of samples while preserving privacy. Moreover, we introduce a perturbation-based augmentation mechanism that generates synthetic variants of previous data during training, thereby enhancing CL performance. Extensive evaluations on widely used benchmark datasets and settings demonstrate that the proposed algorithm achieves superior performance compared to existing baselines, particularly in scenarios involving large-scale datasets and a high number of tasks.
Paper Structure (46 sections, 14 equations, 3 figures, 10 tables, 6 algorithms)

This paper contains 46 sections, 14 equations, 3 figures, 10 tables, 6 algorithms.

Table of Contents

  1. Introduction
  2. Background & Preliminaries
  3. Online Continual Learning
  4. Prototypical Networks
  5. Dataset Distillation
  6. Method
  7. Efficient Replay Memory
  8. Prototypical Exemplar Generation.
  9. Current Exemplar Alignment.
  10. Previous Exemplar Alignment.
  11. Continual Learning with Prototypical Exemplars
  12. Incremental Feature Extractor
  13. Task-head Learning with Memory
  14. Experimental Settings
  15. Experimental Evaluation
  16. ...and 31 more sections

Figures (3)

  • Figure 1: Illustration of ProtoCore architecture.
  • Figure 2: Last accuracy on tasks observed so far in the test set of S-CIFAR-100 (10, 50 tasks), S-TinyImageNet (20 tasks), and S-ImageNet-1K (100 tasks).
  • Figure 3: t-SNE visualizations of synthetic features generated by ProtoCore on the CIFAR-100 test set. The synthetic data generation process converges within less than 40 epochs, yielding exemplars that closely approximate the real data distribution. $\bigstar$ represents the real prototypes, $\bullet$ represents the embeddings generated by the synthetic exemplars.