Federated Class-Incremental Learning: A Hybrid Approach Using Latent Exemplars and Data-Free Techniques to Address Local and Global Forgetting

TL;DR

The paper tackles Federated Class-Incremental Learning (FCIL), where new classes arrive over time across distributed clients, causing local forgetting and global forgetting under privacy and memory constraints. It introduces Hybrid Replay (HR), a hybrid approach that fuses latent-exemplar replay with data-free synthetic replay via a customized autoencoder, and uses a server-side Lennard-Jones Potential to align class-centroid embeddings in a shared latent space. A formal FCIL framework is provided, including recursive global loss updates $I_{\boldsymbol{\theta}}^{(k+1)}$ and a dual-loss mechanism that controls intra- and inter-client forgetting. Empirical results on multiple FCIL benchmarks show that HR achieves state-of-the-art performance with low memory and compute costs, while preserving privacy, and ablation studies highlight the critical roles of latent exemplars, knowledge distillation, and global replay.

Abstract

Federated Class-Incremental Learning (FCIL) refers to a scenario where a dynamically changing number of clients collaboratively learn an ever-increasing number of incoming tasks. FCIL is known to suffer from local forgetting due to class imbalance at each client and global forgetting due to class imbalance across clients. We develop a mathematical framework for FCIL that formulates local and global forgetting. Then, we propose an approach called Hybrid Rehearsal (HR), which utilizes latent exemplars and data-free techniques to address local and global forgetting, respectively. HR employs a customized autoencoder designed for both data classification and the generation of synthetic data. To determine the embeddings of new tasks for all clients in the latent space of the encoder, the server uses the Lennard-Jones Potential formulations. Meanwhile, at the clients, the decoder decodes the stored low-dimensional latent space exemplars back to the high-dimensional input space, used to address local forgetting. To overcome global forgetting, the decoder generates synthetic data. Furthermore, our mathematical framework proves that our proposed approach HR can, in principle, tackle the two local and global forgetting challenges. In practice, extensive experiments demonstrate that while preserving privacy, our proposed approach outperforms the state-of-the-art baselines on multiple FCIL benchmarks with low compute and memory footprints.

Figures (5)

• Figure 1: Local and global forgetting occur due to class imbalance at both the local (client-level) and global (system-wide) scales. Whereas local forgetting refers to the combination of sub-optimal losses for the previous tasks and intra-client task confusion, global forgetting involves inter-client task confusion.
• Figure 2: Our HR approach, except for Task 1, consists of two key phases for handling tasks in FCIL. For Task 1, the autoencoder is trained, and the compact latent representations $M_1$ are stored. For subsequent tasks $h$, in the Model Update Phase, compact representations from previous tasks $M_{1:h-1}$ are decoded, interleaved with the new task's data $D_h$, and used to update the model. In the Memory Update Phase, with the updated model, compact representations for both the old tasks and the new task are computed and stored $M_{1:h}$.
• Figure 3: In the first row, the table reports the impact of the memory size on the final accuracy for the three FCIL configurations. In the second row, the forgetting trends are shown for HR and various baselines. Hybrid replay consistently outperforms the model-based or data-based approaches.
• Figure : HR Algorithm for FCIL at the Server
• Figure : HR Algorithm for FCIL at Clients