Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Prototype Augmented Hypernetworks for Continual Learning

Neil De La Fuente, Maria Pilligua, Daniel Vidal, Albin Soutiff, Cecilia Curreli, Daniel Cremers, Andrey Barsky

TL;DR

Prototype-Augmented Hypernetworks (PAH) address catastrophic forgetting in task-incremental continual learning by generating task-specific classifier heads on demand through a hypernetwork conditioned on learnable prototypes. Prototypes act as a compact, semantically informed task embedding that, together with dual knowledge-distillation losses, stabilizes representations without storing per-task heads or replay buffers. Empirical results on Split-CIFAR100 and TinyImageNet show state-of-the-art accuracy with minimal forgetting, and ablations highlight the importance of semantic prototype initialization, a 10×10 prototype grid, and prototype-focused distillation. PAH offers a scalable, memory-efficient solution to continual learning with strong practical impact for dynamic environments.

Abstract

Continual learning (CL) aims to learn a sequence of tasks without forgetting prior knowledge, but gradient updates for a new task often overwrite the weights learned earlier, causing catastrophic forgetting (CF). We propose Prototype-Augmented Hypernetworks (PAH), a framework where a single hypernetwork, conditioned on learnable task prototypes, dynamically generates task-specific classifier heads on demand. To mitigate forgetting, PAH combines cross-entropy with dual distillation losses, one to align logits and another to align prototypes, ensuring stable feature representations across tasks. Evaluations on Split-CIFAR100 and TinyImageNet demonstrate that PAH achieves state-of-the-art performance, reaching 74.5 % and 63.7 % accuracy with only 1.7 % and 4.4 % forgetting, respectively, surpassing prior methods without storing samples or heads.

Prototype Augmented Hypernetworks for Continual Learning

TL;DR

Prototype-Augmented Hypernetworks (PAH) address catastrophic forgetting in task-incremental continual learning by generating task-specific classifier heads on demand through a hypernetwork conditioned on learnable prototypes. Prototypes act as a compact, semantically informed task embedding that, together with dual knowledge-distillation losses, stabilizes representations without storing per-task heads or replay buffers. Empirical results on Split-CIFAR100 and TinyImageNet show state-of-the-art accuracy with minimal forgetting, and ablations highlight the importance of semantic prototype initialization, a 10×10 prototype grid, and prototype-focused distillation. PAH offers a scalable, memory-efficient solution to continual learning with strong practical impact for dynamic environments.

Abstract

Continual learning (CL) aims to learn a sequence of tasks without forgetting prior knowledge, but gradient updates for a new task often overwrite the weights learned earlier, causing catastrophic forgetting (CF). We propose Prototype-Augmented Hypernetworks (PAH), a framework where a single hypernetwork, conditioned on learnable task prototypes, dynamically generates task-specific classifier heads on demand. To mitigate forgetting, PAH combines cross-entropy with dual distillation losses, one to align logits and another to align prototypes, ensuring stable feature representations across tasks. Evaluations on Split-CIFAR100 and TinyImageNet demonstrate that PAH achieves state-of-the-art performance, reaching 74.5 % and 63.7 % accuracy with only 1.7 % and 4.4 % forgetting, respectively, surpassing prior methods without storing samples or heads.
Paper Structure (17 sections, 6 equations, 1 figure, 2 tables)

This paper contains 17 sections, 6 equations, 1 figure, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Problem Statement
  4. Methodology
  5. a) Main Network
  6. b) Hypernetwork
  7. c) Task Embedding
  8. Loss Functions
  9. Hard Loss Main ($L_{hm}$):
  10. Soft Loss Main ($L_{sm}$):
  11. Soft Loss Prototypes ($L_{sp}$):
  12. Experiments
  13. Datasets
  14. Baselines
  15. Results
  16. ...and 2 more sections

Figures (1)

  • Figure 1: Overview of the Prototype-Augmented Hypernetworks ($PAH$) architecture at task k. The hypernetwork generates task-specific classifier weights conditioned on learnable task prototypes $e_k$, which encode task-specific information. This design eliminates the need to store separate classifier heads for each task and effectively addresses catastrophic forgetting through knowledge distillation.