Federated Continual Learning Goes Online: Uncertainty-Aware Memory Management for Vision Tasks and Beyond

TL;DR

This work tackles Federated Continual Learning in online data-stream settings, where data arrive as mini-batches that can only be processed once. It introduces Online-FCL (O-FCL), a memory-based framework that uses Bregman Information-based predictive uncertainty to curate per-client memory buffers, enabling effective replay without offline data stores. The approach demonstrates reduced catastrophic forgetting and competitive accuracy across vision, medical imaging, and NLP tasks, while maintaining privacy and favorable communication efficiency relative to generator-based baselines. The method’s modality-agnostic memory strategy and online server coordination offer practical benefits for edge deployments and real-world CF scenarios.

Abstract

Given the ability to model more realistic and dynamic problems, Federated Continual Learning (FCL) has been increasingly investigated recently. A well-known problem encountered in this setting is the so-called catastrophic forgetting, for which the learning model is inclined to focus on more recent tasks while forgetting the previously learned knowledge. The majority of the current approaches in FCL propose generative-based solutions to solve said problem. However, this setting requires multiple training epochs over the data, implying an offline setting where datasets are stored locally and remain unchanged over time. Furthermore, the proposed solutions are tailored for vision tasks solely. To overcome these limitations, we propose a new approach to deal with different modalities in the online scenario where new data arrive in streams of mini-batches that can only be processed once. To solve catastrophic forgetting, we propose an uncertainty-aware memory-based approach. Specifically, we suggest using an estimator based on the Bregman Information (BI) to compute the model's variance at the sample level. Through measures of predictive uncertainty, we retrieve samples with specific characteristics, and - by retraining the model on such samples - we demonstrate the potential of this approach to reduce the forgetting effect in realistic settings while maintaining data confidentiality and competitive communication efficiency compared to state-of-the-art approaches.

Figures (5)

• Figure 1: Schematic overview of the proposed online-FCL scenario. Each client $\mathcal{C}_k$ receives their data as a stream of batches. Each batch $b_k^t$ is used for training and for updating the local memory $\mathcal{M}_k$. Once the current batch is processed, the client receives a new batch and continues their training without the possibility of retraining on previously seen batches.
• Figure 2: Illustration of the different aspects of uncertainty captured by model confidence scores and Bregman Information (BI). Close to the decision boundary we can notice how, due to the high aleatoric uncertainty, data points have low-confidence scores. Contrarily, due to high density of observed data, there is a low uncertainty about the data generating process resulting in a low BI.
• Figure 3: (left) Average accuracy per task in comparison with FL (FedAvg) and generative-based FCL baselines (MFCL); (right) Relative CF improvement over the CL baseline (ER).
• Figure 4: Augmentation set used sequentially for the calculation of uncertainty in the experiments.
• Figure 5: Synthetic images generated by MFCL babakniya2024data in the online setting.