Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Lean Clients, Full Accuracy: Hybrid Zeroth- and First-Order Split Federated Learning

Zhoubin Kou, Zihan Chen, Jing Yang, Cong Shen

TL;DR

HERON-SFL tackles the high client-side memory and computation costs in Split Federated Learning by placing zeroth-order optimization on the resource-constrained client while preserving first-order training on the server. The framework leverages an auxiliary network to decouple client updates, enabling local ZO updates without backpropagation or activation caching, and achieves a convergence rate of $O(1/\sqrt{T})$ under a low effective rank assumption, effectively making performance largely independent of model dimensionality. Empirically, it matches state-of-the-art auxiliary-network-based FO SFL in accuracy on ResNet-18 CIFAR-10 and language-model fine-tuning tasks, while reducing peak client memory by up to $64\%$ and per-step FLOPs by up to $33\%$, with comparable communication. The results demonstrate that significant on-device resource savings are possible without sacrificing performance, broadening the range of models that can be trained or adapted on edge devices.

Abstract

Split Federated Learning (SFL) enables collaborative training between resource-constrained edge devices and a compute-rich server. Communication overhead is a central issue in SFL and can be mitigated with auxiliary networks. Yet, the fundamental client-side computation challenge remains, as back-propagation requires substantial memory and computation costs, severely limiting the scale of models that edge devices can support. To enable more resource-efficient client computation and reduce the client-server communication, we propose HERON-SFL, a novel hybrid optimization framework that integrates zeroth-order (ZO) optimization for local client training while retaining first-order (FO) optimization on the server. With the assistance of auxiliary networks, ZO updates enable clients to approximate local gradients using perturbed forward-only evaluations per step, eliminating memory-intensive activation caching and avoiding explicit gradient computation in the traditional training process. Leveraging the low effective rank assumption, we theoretically prove that HERON-SFL's convergence rate is independent of model dimensionality, addressing a key scalability concern common to ZO algorithms. Empirically, on ResNet training and language model (LM) fine-tuning tasks, HERON-SFL matches benchmark accuracy while reducing client peak memory by up to 64% and client-side compute cost by up to 33% per step, substantially expanding the range of models that can be trained or adapted on resource-limited devices.

Lean Clients, Full Accuracy: Hybrid Zeroth- and First-Order Split Federated Learning

TL;DR

HERON-SFL tackles the high client-side memory and computation costs in Split Federated Learning by placing zeroth-order optimization on the resource-constrained client while preserving first-order training on the server. The framework leverages an auxiliary network to decouple client updates, enabling local ZO updates without backpropagation or activation caching, and achieves a convergence rate of under a low effective rank assumption, effectively making performance largely independent of model dimensionality. Empirically, it matches state-of-the-art auxiliary-network-based FO SFL in accuracy on ResNet-18 CIFAR-10 and language-model fine-tuning tasks, while reducing peak client memory by up to and per-step FLOPs by up to , with comparable communication. The results demonstrate that significant on-device resource savings are possible without sacrificing performance, broadening the range of models that can be trained or adapted on edge devices.

Abstract

Split Federated Learning (SFL) enables collaborative training between resource-constrained edge devices and a compute-rich server. Communication overhead is a central issue in SFL and can be mitigated with auxiliary networks. Yet, the fundamental client-side computation challenge remains, as back-propagation requires substantial memory and computation costs, severely limiting the scale of models that edge devices can support. To enable more resource-efficient client computation and reduce the client-server communication, we propose HERON-SFL, a novel hybrid optimization framework that integrates zeroth-order (ZO) optimization for local client training while retaining first-order (FO) optimization on the server. With the assistance of auxiliary networks, ZO updates enable clients to approximate local gradients using perturbed forward-only evaluations per step, eliminating memory-intensive activation caching and avoiding explicit gradient computation in the traditional training process. Leveraging the low effective rank assumption, we theoretically prove that HERON-SFL's convergence rate is independent of model dimensionality, addressing a key scalability concern common to ZO algorithms. Empirically, on ResNet training and language model (LM) fine-tuning tasks, HERON-SFL matches benchmark accuracy while reducing client peak memory by up to 64% and client-side compute cost by up to 33% per step, substantially expanding the range of models that can be trained or adapted on resource-limited devices.
Paper Structure (36 sections, 7 theorems, 65 equations, 7 figures, 3 tables)

This paper contains 36 sections, 7 theorems, 65 equations, 7 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Split Federated Learning
  4. Zeroth-Order Optimization for DML
  5. Preliminaries
  6. SFL with Auxiliary Network
  7. Zeroth-Order Gradient Estimator
  8. Proposed Algorithm: HERON-SFL
  9. Theoretical Analysis
  10. Convergence Analysis
  11. Client-side Resource Cost Analysis
  12. Communication Load
  13. Peak Memory
  14. FLOPs
  15. Experiments
  16. ...and 21 more sections

Key Result

Theorem 1

Under Assumptions assumption:smoothness--assumption:distribution_drift, if the client learning rate satisfies $\eta_c \leq \{\frac{1}{3Lh}, \frac{2}{NLh^2}, \frac{N}{72L}\}$, and is chosen as $\eta_c=\mathcal{O}(\sqrt{{({NB)}/{(dhT)}}})$ while the server learning rate is set to $\eta_s = \mathcal{O}

Figures (7)

  • Figure 1: The proposed HERON-SFL algorithm.
  • Figure 2: ResNet-18 test accuracy vs. communication rounds on CIFAR-10 for IID (left) and non-IID (right) distributions.
  • Figure 3: Test accuracy of different SFL algorithms on CIFAR-10 using a ResNet-18 model. (a) Impact of data heterogeneity under varying Dirichlet $\alpha$ values. (b) Client scalability under different total numbers of clients. (c) Performance under different fractions of participating clients per round.
  • Figure 4: Ablation study on local ZO training hyperparameters using ResNet-18 on CIFAR-10 under an IID setting with ten clients, with all experiments using the same auxiliary model implemented as a single linear layer. Client Size 1 denotes the first convolutional layer and one residual block on the client, while Client Size 2 places three residual blocks on the client. (left) Test accuracy under different perturbation step lengths $\mu$. (right) Test accuracy under different perturbation counts per epoch.
  • Figure 5: GPT2 perplexity curves vs. Communication Volume on E2E for small (left) and medium (right) models.
  • ...and 2 more figures

Theorems & Definitions (17)

  • Definition 1: Gaussian Smoothed Function with Unit-Sphere Normalization
  • Remark 1
  • Theorem 1: Convergence rate of HERON-SFL in the i.i.d. setting
  • Remark 2
  • Theorem 2: Convergence rate of HERON-SFL with Low Effective Rank Assumption
  • Remark 3
  • Remark 4
  • Lemma 1: Gradient and Smoothness for Gaussian Smoothed Functions nesterov2017random
  • Lemma 2: Bound on the Second Moment of the ZO Estimator
  • proof
  • ...and 7 more