Towards Straggler-Resilient Split Federated Learning: An Unbalanced Update Approach

TL;DR

This work tackles the straggler bottleneck in Split Federated Learning by introducing MU-SplitFed, which decouples training progress from slow clients via unbalanced server updates of length $\tau$ and Zeroth-Order optimization on clients. The authors prove nonconvex convergence with a linear speedup in $\tau$ (and in $\tau M$ for multiple clients) under standard smoothness, variance, and heterogeneity assumptions, yielding a convergence rate of $O(\sqrt{d/({\tau T})})$ (and $O(\sqrt{d/({\tau T M})})$ for the federated case). Empirically, MU-SplitFed outperforms baselines under stragglers, reduces communication rounds, and achieves substantial memory savings on edge devices, with demonstrated applicability to LLM fine-tuning. These results suggest a practical, scalable approach to accelerate SFL in heterogeneous environments while mitigating straggler effects and memory constraints. The framework also provides architectural guidelines on how to align server-side depth with the chosen $\tau$ to maximize speedups.

Abstract

Split Federated Learning (SFL) enables scalable training on edge devices by combining the parallelism of Federated Learning (FL) with the computational offloading of Split Learning (SL). Despite its great success, SFL suffers significantly from the well-known straggler issue in distributed learning systems. This problem is exacerbated by the dependency between Split Server and clients: the Split Server side model update relies on receiving activations from clients. Such synchronization requirement introduces significant time latency, making straggler a critical bottleneck to the scalability and efficiency of the system. To mitigate this problem, we propose MU-SplitFed, a straggler-resilient SFL algorithm in zeroth-order optimization that decouples training progress from straggler delays via a simple yet effective unbalanced update mechanism. By enabling the server to perform $τ$ local updates per client round, MU-SplitFed achieves a convergence rate of $O(\sqrt{d/(τT)})$ for non-convex objectives, demonstrating a linear speedup of $τ$ in communication rounds. Experiments demonstrate that MU-SplitFed consistently outperforms baseline methods with the presence of stragglers and effectively mitigates their impact through adaptive tuning of $τ$. The code for this project is available at https://github.com/Johnny-Zip/MU-SplitFed.

Figures (4)

• Figure 1: Overview of MU-SplitFed. The global model $\textbf{x}$ is split at the cutting layers into two parts: client-side model $x_c$ and server-side model $x_s$. Each client $m$ trains its local copy $x_{c,m}$ while the Split Server performs $\tau$ local updates on $x_{s,m}$ using the latest embedding, without waiting for the client to finish. At the end of each global round, the Fed Server aggregates all client-side models, and the Split Server averages all the server-side models to form the updated global model.
• Figure 2: Performance Under Stragglers, where we set $\tau = 2$ for MU-SplitFed.
• Figure 3: Interaction between cut layer $L_c$ and server iteration $\tau$.
• Figure 4: Comparison of peak memory cost for different methods for fine-tuning LLM.

Theorems & Definitions (14)

• Theorem 4.1
• Corollary 4.2
• Theorem 4.3
• Corollary 4.4
• Lemma B.1
• Remark B.1
• Lemma B.2: Decomposition
• Theorem C.1
• Lemma C.2: Bounds on the variance of Zeroth-order Gradient
• Lemma C.3: Bounds on the norm of the Zeroth-order gradient estimator