Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rehearsal-Free Continual Federated Learning with Synergistic Synaptic Intelligence

Yichen Li, Yuying Wang, Haozhao Wang, Yining Qi, Tianzhe Xiao, Ruixuan Li

TL;DR

This work tackles Continual Federated Learning (CFL) under data heterogeneity and privacy constraints by avoiding data rehearsal. It introduces FedSSI, a regularization-based method that tailors Synaptic Intelligence (SI) through a Personalized Surrogate Model (PSM) to integrate local and global information, mitigating forgetting without replay. The authors provide theoretical insights (Proposition 1 and Theorem 1) on balancing global/local influence and convergence, and show extensive experiments across six datasets and two incremental task scenarios where FedSSI achieves up to 12.47% higher final accuracy than baselines. The approach offers a resource-efficient, privacy-preserving alternative for CFL applicable to real-world, heterogeneous, and device-constrained deployments.

Abstract

Continual Federated Learning (CFL) allows distributed devices to collaboratively learn novel concepts from continuously shifting training data while avoiding knowledge forgetting of previously seen tasks. To tackle this challenge, most current CFL approaches rely on extensive rehearsal of previous data. Despite effectiveness, rehearsal comes at a cost to memory, and it may also violate data privacy. Considering these, we seek to apply regularization techniques to CFL by considering their cost-efficient properties that do not require sample caching or rehearsal. Specifically, we first apply traditional regularization techniques to CFL and observe that existing regularization techniques, especially synaptic intelligence, can achieve promising results under homogeneous data distribution but fail when the data is heterogeneous. Based on this observation, we propose a simple yet effective regularization algorithm for CFL named FedSSI, which tailors the synaptic intelligence for the CFL with heterogeneous data settings. FedSSI can not only reduce computational overhead without rehearsal but also address the data heterogeneity issue. Extensive experiments show that FedSSI achieves superior performance compared to state-of-the-art methods.

Rehearsal-Free Continual Federated Learning with Synergistic Synaptic Intelligence

TL;DR

This work tackles Continual Federated Learning (CFL) under data heterogeneity and privacy constraints by avoiding data rehearsal. It introduces FedSSI, a regularization-based method that tailors Synaptic Intelligence (SI) through a Personalized Surrogate Model (PSM) to integrate local and global information, mitigating forgetting without replay. The authors provide theoretical insights (Proposition 1 and Theorem 1) on balancing global/local influence and convergence, and show extensive experiments across six datasets and two incremental task scenarios where FedSSI achieves up to 12.47% higher final accuracy than baselines. The approach offers a resource-efficient, privacy-preserving alternative for CFL applicable to real-world, heterogeneous, and device-constrained deployments.

Abstract

Continual Federated Learning (CFL) allows distributed devices to collaboratively learn novel concepts from continuously shifting training data while avoiding knowledge forgetting of previously seen tasks. To tackle this challenge, most current CFL approaches rely on extensive rehearsal of previous data. Despite effectiveness, rehearsal comes at a cost to memory, and it may also violate data privacy. Considering these, we seek to apply regularization techniques to CFL by considering their cost-efficient properties that do not require sample caching or rehearsal. Specifically, we first apply traditional regularization techniques to CFL and observe that existing regularization techniques, especially synaptic intelligence, can achieve promising results under homogeneous data distribution but fail when the data is heterogeneous. Based on this observation, we propose a simple yet effective regularization algorithm for CFL named FedSSI, which tailors the synaptic intelligence for the CFL with heterogeneous data settings. FedSSI can not only reduce computational overhead without rehearsal but also address the data heterogeneity issue. Extensive experiments show that FedSSI achieves superior performance compared to state-of-the-art methods.

Paper Structure

This paper contains 19 sections, 9 equations, 4 figures, 12 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Preliminaries
  4. Methodology
  5. Observations
  6. FedSSI: Improved Regularization Technique with Synergism for CFL
  7. Analytical Understanding of the Personalized Surrogate Model in FedSSI
  8. Experiments
  9. Experiment Setup
  10. Performance Overview
  11. Conclusion
  12. Datasets
  13. Baselines
  14. Configurations
  15. Additional Results
  16. ...and 4 more sections

Figures (4)

  • Figure 1: Performance comparisons of regularization-based CFL methods on CIFAR10 and Digit10 datasets with IID data.
  • Figure 2: Performance comparisons of aforementioned methods on CIFAR10 and Digit10 datasets with Non-IID data.
  • Figure 3: Performance w.r.t data heterogeneity $\alpha$ for four datasets.
  • Figure 4: Performance w.r.t number of incremental tasks $n$ for two class-incremental datasets