Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerating Wireless Distributed Learning via Hybrid Split and Federated Learning Optimization

Kun Guo, Xuefei Li, Xijun Wang, Howard H. Yang, Wei Feng, Tony Q. S. Quek

TL;DR

This work tackles accelerating wireless distributed learning by enabling Hybrid Split and Federated Learning (HSFL), where devices can operate in FL or SL mode in parallel. It develops a convergence-informed delay-minimization framework that jointly optimizes learning mode selection, model splitting, bandwidth allocation, and batch sizes, solved via a two-stage block-coordinate method with a batch-size rounding step. The proposed approach balances per-round delay with the number of rounds to convergence, and empirical results show significant reductions in overall learning delay to reach a target accuracy, especially under data heterogeneity. The methods enable practical HSFL deployment in resource-constrained wireless networks, providing a principled way to exploit both low-latency FL and high-accuracy SL within a single framework.

Abstract

Federated learning (FL) and split learning (SL) are two effective distributed learning paradigms in wireless networks, enabling collaborative model training across mobile devices without sharing raw data. While FL supports low-latency parallel training, it may converge to less accurate model. In contrast, SL achieves higher accuracy through sequential training but suffers from increased delay. To leverage the advantages of both, hybrid split and federated learning (HSFL) allows some devices to operate in FL mode and others in SL mode. This paper aims to accelerate HSFL by addressing three key questions: 1) How does learning mode selection affect overall learning performance? 2) How does it interact with batch size? 3) How can these hyperparameters be jointly optimized alongside communication and computational resources to reduce overall learning delay? We first analyze convergence, revealing the interplay between learning mode and batch size. Next, we formulate a delay minimization problem and propose a two-stage solution: a block coordinate descent method for a relaxed problem to obtain a locally optimal solution, followed by a rounding algorithm to recover integer batch sizes with near-optimal performance. Experimental results demonstrate that our approach significantly accelerates convergence to the target accuracy compared to existing methods.

Accelerating Wireless Distributed Learning via Hybrid Split and Federated Learning Optimization

TL;DR

This work tackles accelerating wireless distributed learning by enabling Hybrid Split and Federated Learning (HSFL), where devices can operate in FL or SL mode in parallel. It develops a convergence-informed delay-minimization framework that jointly optimizes learning mode selection, model splitting, bandwidth allocation, and batch sizes, solved via a two-stage block-coordinate method with a batch-size rounding step. The proposed approach balances per-round delay with the number of rounds to convergence, and empirical results show significant reductions in overall learning delay to reach a target accuracy, especially under data heterogeneity. The methods enable practical HSFL deployment in resource-constrained wireless networks, providing a principled way to exploit both low-latency FL and high-accuracy SL within a single framework.

Abstract

Federated learning (FL) and split learning (SL) are two effective distributed learning paradigms in wireless networks, enabling collaborative model training across mobile devices without sharing raw data. While FL supports low-latency parallel training, it may converge to less accurate model. In contrast, SL achieves higher accuracy through sequential training but suffers from increased delay. To leverage the advantages of both, hybrid split and federated learning (HSFL) allows some devices to operate in FL mode and others in SL mode. This paper aims to accelerate HSFL by addressing three key questions: 1) How does learning mode selection affect overall learning performance? 2) How does it interact with batch size? 3) How can these hyperparameters be jointly optimized alongside communication and computational resources to reduce overall learning delay? We first analyze convergence, revealing the interplay between learning mode and batch size. Next, we formulate a delay minimization problem and propose a two-stage solution: a block coordinate descent method for a relaxed problem to obtain a locally optimal solution, followed by a rounding algorithm to recover integer batch sizes with near-optimal performance. Experimental results demonstrate that our approach significantly accelerates convergence to the target accuracy compared to existing methods.

Paper Structure

This paper contains 31 sections, 1 theorem, 42 equations, 8 figures, 6 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Wireless Distributed Learning Frameworks
  4. Learning Optimization Methods
  5. Learning Workflow and Delay Model
  6. Learning Workflow
  7. Model Training of FL Devices
  8. Model Training of SL Devices
  9. Model Aggregation of FL and SL Devices
  10. Learning Delay
  11. FL Delay
  12. SL Delay
  13. Convergence Analysis and Problem Formulation
  14. Convergence Analysis
  15. Problem Formulation
  16. ...and 16 more sections

Key Result

Theorem 1

Based on Assumptions ass:lipschitz,ass:lsmooth, and ass:secondMoment, we have where $W_t$ is expanded as Note that, $\gamma_1$, $\gamma_2$, $\gamma_3$, and $\gamma_4$ are positive constants that depend on $\kappa$, $\ell$, $G^2$, $M^2$, and learning rate $\eta_t$. Moreover, $\Phi$ is a constant that captures the heterogeneity of the data distribution among devices, with smaller values indicating

Figures (8)

  • Figure 1: An illustration of distributed learning workflow, where the rectangle represents the model layer: (a) FL workflow; (b) SL workflow.
  • Figure 2: Convergence of Algorithm \ref{['alg:alternating']}.
  • Figure 3: Near-optimality of Algorithm \ref{['alg:alternating']}.
  • Figure 4: Overall learning delay comparison across different $(\rho_1,\rho_2^{\prime})$ settings.
  • Figure 5: Impact of $\rho_1$ on learning performance: (a) Overall learning delay and number of rounds to convergence vs. $\rho_1$; (b) Average number of SL devices and average total batch size per round vs. $\rho_1$.
  • ...and 3 more figures

Theorems & Definitions (5)

  • Theorem 1
  • Remark 1
  • Remark 2
  • Remark 3
  • Remark 4