Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KoReA-SFL: Knowledge Replay-based Split Federated Learning Against Catastrophic Forgetting

Zeke Xia, Ming Hu, Dengke Yan, Ruixuan Liu, Anran Li, Xiaofei Xie, Mingsong Chen

TL;DR

KoReA-SFL addresses non-IID data-induced gradient divergence and catastrophic forgetting in Split Federated Learning by maintaining multiple branch server-side and client-side portions with a shared master model for knowledge sharing. It introduces a knowledge replay mechanism that solicitously samples underrepresented category features from assistant devices based on per-branch data-distribution scores, combined with an adaptive sampling proportion governed by the Federated Gradient Norm. The approach is supported by a convergence analysis under standard assumptions and empirical results showing substantial accuracy gains (up to about 23% or more in challenging non-IID settings) across diverse datasets and models, along with ablation studies validating the contributions of knowledge replay and adaptive sampling. Overall, KoReA-SFL demonstrates improved robustness, stability, and generalization in resource-constrained, heterogeneous federated environments, enabling more effective SFL deployments.

Abstract

Although Split Federated Learning (SFL) is good at enabling knowledge sharing among resource-constrained clients, it suffers from the problem of low training accuracy due to the neglect of data heterogeneity and catastrophic forgetting. To address this issue, we propose a novel SFL approach named KoReA-SFL, which adopts a multi-model aggregation mechanism to alleviate gradient divergence caused by heterogeneous data and a knowledge replay strategy to deal with catastrophic forgetting. Specifically, in KoReA-SFL cloud servers (i.e., fed server and main server) maintain multiple branch model portions rather than a global portion for local training and an aggregated master-model portion for knowledge sharing among branch portions. To avoid catastrophic forgetting, the main server of KoReA-SFL selects multiple assistant devices for knowledge replay according to the training data distribution of each server-side branch-model portion. Experimental results obtained from non-IID and IID scenarios demonstrate that KoReA-SFL significantly outperforms conventional SFL methods (by up to 23.25\% test accuracy improvement).

KoReA-SFL: Knowledge Replay-based Split Federated Learning Against Catastrophic Forgetting

TL;DR

KoReA-SFL addresses non-IID data-induced gradient divergence and catastrophic forgetting in Split Federated Learning by maintaining multiple branch server-side and client-side portions with a shared master model for knowledge sharing. It introduces a knowledge replay mechanism that solicitously samples underrepresented category features from assistant devices based on per-branch data-distribution scores, combined with an adaptive sampling proportion governed by the Federated Gradient Norm. The approach is supported by a convergence analysis under standard assumptions and empirical results showing substantial accuracy gains (up to about 23% or more in challenging non-IID settings) across diverse datasets and models, along with ablation studies validating the contributions of knowledge replay and adaptive sampling. Overall, KoReA-SFL demonstrates improved robustness, stability, and generalization in resource-constrained, heterogeneous federated environments, enabling more effective SFL deployments.

Abstract

Although Split Federated Learning (SFL) is good at enabling knowledge sharing among resource-constrained clients, it suffers from the problem of low training accuracy due to the neglect of data heterogeneity and catastrophic forgetting. To address this issue, we propose a novel SFL approach named KoReA-SFL, which adopts a multi-model aggregation mechanism to alleviate gradient divergence caused by heterogeneous data and a knowledge replay strategy to deal with catastrophic forgetting. Specifically, in KoReA-SFL cloud servers (i.e., fed server and main server) maintain multiple branch model portions rather than a global portion for local training and an aggregated master-model portion for knowledge sharing among branch portions. To avoid catastrophic forgetting, the main server of KoReA-SFL selects multiple assistant devices for knowledge replay according to the training data distribution of each server-side branch-model portion. Experimental results obtained from non-IID and IID scenarios demonstrate that KoReA-SFL significantly outperforms conventional SFL methods (by up to 23.25\% test accuracy improvement).
Paper Structure (13 sections, 1 theorem, 8 equations, 9 figures, 2 tables, 1 algorithm)

This paper contains 13 sections, 1 theorem, 8 equations, 9 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Our KoReA-SFL Approach
  4. Overview
  5. Implementation
  6. Knowledge Replay Strategy
  7. Adaptive Sample Proportion
  8. Convergence Analysis
  9. Experimental Results
  10. Experimental Settings
  11. Performance Comparison
  12. Impacts of Different Configurations
  13. Conclusion

Key Result

Lemma 1

Let $w_{r}^i= \alpha v_{r}^i + (1-\alpha)\overline{v}_{r}$, $\alpha\in [0,1]$, and $\overline{w}_r = \sum_{i=1}^N w_{r}^i$. We have where $w^\star$ is the optimal parameters for the global loss function $F(\cdot)$. In other words, $\forall w, F^\star\leq F(w)$, where $F^\star$ denotes $F(w^\star)$.

Figures (9)

  • Figure 1: Motivation of our approach.
  • Figure 2: Framework and workflow of our approach.
  • Figure 3: Comparison of communication overhead.
  • Figure 4: Comparison of loss landscapes.
  • Figure 5: Learning curves for the different configurations.
  • ...and 4 more figures

Theorems & Definitions (1)

  • Lemma 1