Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GAPSL: A Gradient-Aligned Parallel Split Learning on Heterogeneous Data

Zheng Lin, Ons Aouedi, Wei Ni, Symeon Chatzinotas, Xianhao Chen

Abstract

The increasing complexity of neural networks poses significant challenges for democratizing FL on resource?constrained client devices. Parallel split learning (PSL) has emerged as a promising solution by offloading substantial computing workload to a server via model partitioning, shrinking client-side computing load, and eliminating the client-side model aggregation for reduced communication and deployment costs. Since PSL is aggregation-free, it suffers from severe training divergence stemming from gradient directional inconsistency across clients. To address this challenge, we propose GAPSL, a gradient-aligned PSL framework that comprises two key components: leader gradient identification (LGI) and gradient direction alignment (GDA). LGI dynamically selects a set of directionally consistent client gradients to construct a leader gradient that captures the global convergence trend. GDA employs a direction-aware regularization to align each client's gradient with the leader gradient, thereby mitigating inter-device gradient directional inconsistency and enhancing model convergence. We evaluate GAPSL on a prototype computing testbed. Extensive experiments demonstrate that GAPSL consistently outperforms state-of-the-art benchmarks in training accuracy and latency.

GAPSL: A Gradient-Aligned Parallel Split Learning on Heterogeneous Data

Abstract

The increasing complexity of neural networks poses significant challenges for democratizing FL on resource?constrained client devices. Parallel split learning (PSL) has emerged as a promising solution by offloading substantial computing workload to a server via model partitioning, shrinking client-side computing load, and eliminating the client-side model aggregation for reduced communication and deployment costs. Since PSL is aggregation-free, it suffers from severe training divergence stemming from gradient directional inconsistency across clients. To address this challenge, we propose GAPSL, a gradient-aligned PSL framework that comprises two key components: leader gradient identification (LGI) and gradient direction alignment (GDA). LGI dynamically selects a set of directionally consistent client gradients to construct a leader gradient that captures the global convergence trend. GDA employs a direction-aware regularization to align each client's gradient with the leader gradient, thereby mitigating inter-device gradient directional inconsistency and enhancing model convergence. We evaluate GAPSL on a prototype computing testbed. Extensive experiments demonstrate that GAPSL consistently outperforms state-of-the-art benchmarks in training accuracy and latency.
Paper Structure (33 sections, 8 equations, 21 figures, 3 tables, 2 algorithms)

This paper contains 33 sections, 8 equations, 21 figures, 3 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Challenge and Motivation
  3. Lack of Client-side Model Aggregation
  4. Gradient Directional Inconsistency
  5. System Design
  6. Overview
  7. Leader Gradient Identification
  8. Gradient Consistency Scoring
  9. Adaptive Gradient Selection
  10. Leader Gradient Construction
  11. Gradient Direction Alignment
  12. Angular Deviation Computation
  13. Adaptive Angular Thresholding
  14. Gradient Direction Regularization
  15. Implementation
  16. ...and 18 more sections

Figures (21)

  • Figure 1: The comparison of SFL, PSL, and GAPSL frameworks, where SFL relies on a parameter server to aggregate client-side models and PSL streamlines the training process by eliminating the client-side model aggregation.
  • Figure 2: The training performance (a) and test accuracy (b) of SFL and PSL on CIFAR-10 using VGG-16.
  • Figure 3: The test accuracy (a) and average angular deviation (b) versus degree of data heterogeneity on CIFAR-10 using VGG-16.
  • Figure 4: An overview of GAPSL architecture.
  • Figure 5: The PSL performance of training a model with the gradient from a single client device only (discarding others' gradients) on CIFAR-10 using VGG-16 under the non-IID setting.
  • ...and 16 more figures