Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Continual Learning via Knowledge Fusion: A Survey

Xin Yang, Hao Yu, Xin Gao, Hao Wang, Junbo Zhang, Tianrui Li

TL;DR

The paper surveys Federated Continual Learning (FCL) as the fusion of privacy-preserving federated learning and sequential task learning, introducing spatial-temporal catastrophic forgetting as a core challenge. It proposes two generic FCL frameworks—synchronous and asynchronous—and a seven-form taxonomy of knowledge fusion methods to fuse knowledge across clients and tasks. Through analysis and experiments with FedAvg on CIFAR-100, it demonstrates how data heterogeneity and task arrangement drive forgetting and highlights the necessity of effective knowledge fusion. The work outlines future directions for trustworthy FCL, convergence efficiency, and integration with broader AI systems such as large language models, underscoring the practical impact of robust, privacy-preserving continual learning in distributed environments.

Abstract

Data privacy and silos are nontrivial and greatly challenging in many real-world applications. Federated learning is a decentralized approach to training models across multiple local clients without the exchange of raw data from client devices to global servers. However, existing works focus on a static data environment and ignore continual learning from streaming data with incremental tasks. Federated Continual Learning (FCL) is an emerging paradigm to address model learning in both federated and continual learning environments. The key objective of FCL is to fuse heterogeneous knowledge from different clients and retain knowledge of previous tasks while learning on new ones. In this work, we delineate federated learning and continual learning first and then discuss their integration, i.e., FCL, and particular FCL via knowledge fusion. In summary, our motivations are four-fold: we (1) raise a fundamental problem called ''spatial-temporal catastrophic forgetting'' and evaluate its impact on the performance using a well-known method called federated averaging (FedAvg), (2) integrate most of the existing FCL methods into two generic frameworks, namely synchronous FCL and asynchronous FCL, (3) categorize a large number of methods according to the mechanism involved in knowledge fusion, and finally (4) showcase an outlook on the future work of FCL.

Federated Continual Learning via Knowledge Fusion: A Survey

TL;DR

The paper surveys Federated Continual Learning (FCL) as the fusion of privacy-preserving federated learning and sequential task learning, introducing spatial-temporal catastrophic forgetting as a core challenge. It proposes two generic FCL frameworks—synchronous and asynchronous—and a seven-form taxonomy of knowledge fusion methods to fuse knowledge across clients and tasks. Through analysis and experiments with FedAvg on CIFAR-100, it demonstrates how data heterogeneity and task arrangement drive forgetting and highlights the necessity of effective knowledge fusion. The work outlines future directions for trustworthy FCL, convergence efficiency, and integration with broader AI systems such as large language models, underscoring the practical impact of robust, privacy-preserving continual learning in distributed environments.

Abstract

Data privacy and silos are nontrivial and greatly challenging in many real-world applications. Federated learning is a decentralized approach to training models across multiple local clients without the exchange of raw data from client devices to global servers. However, existing works focus on a static data environment and ignore continual learning from streaming data with incremental tasks. Federated Continual Learning (FCL) is an emerging paradigm to address model learning in both federated and continual learning environments. The key objective of FCL is to fuse heterogeneous knowledge from different clients and retain knowledge of previous tasks while learning on new ones. In this work, we delineate federated learning and continual learning first and then discuss their integration, i.e., FCL, and particular FCL via knowledge fusion. In summary, our motivations are four-fold: we (1) raise a fundamental problem called ''spatial-temporal catastrophic forgetting'' and evaluate its impact on the performance using a well-known method called federated averaging (FedAvg), (2) integrate most of the existing FCL methods into two generic frameworks, namely synchronous FCL and asynchronous FCL, (3) categorize a large number of methods according to the mechanism involved in knowledge fusion, and finally (4) showcase an outlook on the future work of FCL.
Paper Structure (29 sections, 4 equations, 4 figures, 4 tables)

This paper contains 29 sections, 4 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Federated and Continual Learning: An Overview
  3. Federated Learning
  4. Continual Learning
  5. Spatial-Temporal Catastrophic Forgetting
  6. Problem Definition
  7. Performance Evaluation
  8. Settings
  9. Datasets
  10. Metrics
  11. Experimental Results and Analysis
  12. Federated Continual Learning
  13. Synchronous FCL
  14. Asynchronous FCL
  15. Federated Task-Incremental Learning
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Experiment Results. Solid lines represent knowledge retention of clients in time, and dashed lines represent knowledge retention of clients in space. When the data distribution among clients is identical, and the distribution among local tasks is also the same, spatial-temporal catastrophic forgetting does not occur, and the model's performance improves. However, severe performance degradation occurs after aggregation when clients have training data with different classes. Additionally, when the local task sequence follows a class-incremental pattern, the model forgets all knowledge from previous tasks, resulting in temporal knowledge retention to zero.
  • Figure 2: Illustration of synchronous federated continual learning scenario, in which all clients share a common task sequence. Different colors represent different tasks in a task sequence. Apparently, "synchronous" means that clients will process the same task in a shared task order, and the progress of all clients is consistent. Synchronous FCL mainly focuses on aggregation.
  • Figure 3: Illustration of Asynchronous FCL. In Asynchronous FCL, the process of fusing knowledge is gradual, which means that once a client has finished training on a task, the server timely fuses it into the global model without waiting for other clients. An ideal Asynchronous FCL can turn aggregation into fusion, allowing much more tolerance of asynchronous clients.
  • Figure 4: Knowledge Fusion in FCL.