Autoencoder-Based Hybrid Replay for Class-Incremental Learning

TL;DR

This work tackles class-incremental learning (CIL) by addressing catastrophic forgetting and task confusion with a novel hybrid replay strategy. It introduces Autoencoder-Based Hybrid Replay (AHR) built on a Hybrid Autoencoder (HAE) that compresses exemplars into a latent space and uses a charged particle system energy minimization framework (CPSEM) with a repulsive force algorithm (RFA) to position class centroids, enabling classification via Euclidean distance in latent space. AHR stores latent exemplars rather than raw data, achieving memory complexity of $\mathcal{O}(0.1 t)$ and compute complexity $\mathcal{O}(t)$, while preserving or surpassing state-of-the-art accuracy across benchmarks. The approach blends exemplar and generative replay advantages, demonstrates strong empirical results, and provides practical memory efficiency for scalable continual learning.

Abstract

In class-incremental learning (CIL), effective incremental learning strategies are essential to mitigate task confusion and catastrophic forgetting, especially as the number of tasks $t$ increases. Current exemplar replay strategies impose $\mathcal{O}(t)$ memory/compute complexities. We propose an autoencoder-based hybrid replay (AHR) strategy that leverages our new hybrid autoencoder (HAE) to function as a compressor to alleviate the requirement for large memory, achieving $\mathcal{O}(0.1 t)$ at the worst case with the computing complexity of $\mathcal{O}(t)$ while accomplishing state-of-the-art performance. The decoder later recovers the exemplar data stored in the latent space, rather than in raw format. Additionally, HAE is designed for both discriminative and generative modeling, enabling classification and replay capabilities, respectively. HAE adopts the charged particle system energy minimization equations and repulsive force algorithm for the incremental embedding and distribution of new class centroids in its latent space. Our results demonstrate that AHR consistently outperforms recent baselines across multiple benchmarks while operating with the same memory/compute budgets. The source code is included in the supplementary material and will be open-sourced upon publication.

Figures (5)

• Figure 1: (a) Usage of RFA for the latent space. (b) Adoption of Euclidean distance during test. (c) HAE for compression and decompression of the dataset for replay.
• Figure 2: Task-based and task-free.
• Figure 3: The impact of the memory size (first row). The required resources to achieve a target performance (second row). The achieved performance for a given compute time (third row).
• Figure 4: Images produced by the decoder at different tasks (for the decoder size of $1.8$M).
• Figure 5: Performances for various decoder/memory sizes.