Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Split Federated Learning Over Heterogeneous Edge Devices: Algorithm and Optimization

Yunrui Sun, Gang Hu, Yinglei Teng, Dunbo Cai

TL;DR

This work proposes the Heterogeneous Split Federated Learning (HSFL) framework, which allows resource-constrained clients to train their personalized client-side models in parallel, utilizing different cut layers, and designs a latency minimization problem that optimizes computational and transmission resources jointly.

Abstract

Split Learning (SL) is a promising collaborative machine learning approach, enabling resource-constrained devices to train models without sharing raw data, while reducing computational load and preserving privacy simultaneously. However, current SL algorithms face limitations in training efficiency and suffer from prolonged latency, particularly in sequential settings, where the slowest device can bottleneck the entire process due to heterogeneous resources and frequent data exchanges between clients and servers. To address these challenges, we propose the Heterogeneous Split Federated Learning (HSFL) framework, which allows resource-constrained clients to train their personalized client-side models in parallel, utilizing different cut layers. Aiming to mitigate the impact of heterogeneous environments and accelerate the training process, we formulate a latency minimization problem that optimizes computational and transmission resources jointly. Additionally, we design a resource allocation algorithm that combines the Sample Average Approximation (SAA), Genetic Algorithm (GA), Lagrangian relaxation and Branch and Bound (B\&B) methods to efficiently solve this problem. Simulation results demonstrate that HSFL outperforms other frameworks in terms of both convergence rate and model accuracy on heterogeneous devices with non-iid data, while the optimization algorithm is better than other baseline methods in reducing latency.

Split Federated Learning Over Heterogeneous Edge Devices: Algorithm and Optimization

TL;DR

This work proposes the Heterogeneous Split Federated Learning (HSFL) framework, which allows resource-constrained clients to train their personalized client-side models in parallel, utilizing different cut layers, and designs a latency minimization problem that optimizes computational and transmission resources jointly.

Abstract

Split Learning (SL) is a promising collaborative machine learning approach, enabling resource-constrained devices to train models without sharing raw data, while reducing computational load and preserving privacy simultaneously. However, current SL algorithms face limitations in training efficiency and suffer from prolonged latency, particularly in sequential settings, where the slowest device can bottleneck the entire process due to heterogeneous resources and frequent data exchanges between clients and servers. To address these challenges, we propose the Heterogeneous Split Federated Learning (HSFL) framework, which allows resource-constrained clients to train their personalized client-side models in parallel, utilizing different cut layers. Aiming to mitigate the impact of heterogeneous environments and accelerate the training process, we formulate a latency minimization problem that optimizes computational and transmission resources jointly. Additionally, we design a resource allocation algorithm that combines the Sample Average Approximation (SAA), Genetic Algorithm (GA), Lagrangian relaxation and Branch and Bound (B\&B) methods to efficiently solve this problem. Simulation results demonstrate that HSFL outperforms other frameworks in terms of both convergence rate and model accuracy on heterogeneous devices with non-iid data, while the optimization algorithm is better than other baseline methods in reducing latency.

Paper Structure

This paper contains 19 sections, 19 equations, 5 figures, 1 table.

Table of Contents

  1. Introduction
  2. HSFL Design
  3. Forward Propagation Stage
  4. Back Propagation Stage
  5. Synchronous Aggregation Stage
  6. System Latency Model
  7. Latency of Forward Propagation Stage
  8. Latency of Back Propagation Stage
  9. Latency of Synchronous Aggregation Stage
  10. Problem Formulation And Solutions
  11. Problem Formulation
  12. Problem Solution
  13. Long-timescale Variables Solution
  14. Short-Timescale Variables Solution
  15. Simulation And Performance Evaluation
  16. ...and 4 more sections

Figures (5)

  • Figure 1: Framework (a) and single-round training latency (b) of HSFL. The blank in (b) means idle waiting for multi-clients.
  • Figure 2: Flow of SAA&GA-based algorithm.
  • Figure 3: Convergence speed comparison on HAM10000.
  • Figure 4: Single-round latency on different baselines.
  • Figure 5: Single-round latency with different device heterogeneity.