Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ELSA: Efficient LLM-Centric Split Aggregation for Privacy-Aware Hierarchical Federated Learning over Resource-Constrained Edge Networks

Xiaohong Yang, Tong Xie, Minghui Liwang, Chikai Shang, Yang Lu, Zhenzhen Jiao, Liqun Fu, Seyyedali Hosseinalipour

TL;DR

This work tackles edge-LLM fine-tuning under resource limits, non-IID data, and privacy constraints. It presents ELSA, a hierarchical, split-learning framework that fuses SL and HFL with behavior-aware clustering, a tripartite LLM split, and semantic subspace orthogonal perturbation (SS-OP) with computational sketches to preserve privacy and reduce communication. Across eight NLP tasks, ELSA demonstrates faster convergence and higher final accuracy while delivering substantial communication savings, including up to about $4.7$-fold throughput gains under suitable compression ratios. The approach offers a scalable, privacy-aware path for edge-adaptive LLM fine-tuning and can be extended to ultra-large models with adaptive splitting and compression strategies.

Abstract

Training large language models (LLMs) at the network edge faces fundamental challenges arising from device resource constraints, severe data heterogeneity, and heightened privacy risks. To address these, we propose ELSA (Efficient LLM-centric Split Aggregation), a novel framework that systematically integrates split learning (SL) and hierarchical federated learning (HFL) for distributed LLM fine-tuning over resource-constrained edge networks. ELSA introduces three key innovations. First, it employs a task-agnostic, behavior-aware client clustering mechanism that constructs semantic fingerprints using public probe inputs and symmetric KL divergence, further enhanced by prediction-consistency-based trust scoring and latency-aware edge assignment to jointly address data heterogeneity, client unreliability, and communication constraints. Second, it splits the LLM into three parts across clients and edge servers, with the cloud used only for adapter aggregation, enabling an effective balance between on-device computation cost and global convergence stability. Third, it incorporates a lightweight communication scheme based on computational sketches combined with semantic subspace orthogonal perturbation (SS-OP) to reduce communication overhead while mitigating privacy leakage during model exchanges. Experiments across diverse NLP tasks demonstrate that ELSA consistently outperforms state-of-the-art methods in terms of adaptability, convergence behavior, and robustness, establishing a scalable and privacy-aware solution for edge-side LLM fine-tuning under resource constraints.

ELSA: Efficient LLM-Centric Split Aggregation for Privacy-Aware Hierarchical Federated Learning over Resource-Constrained Edge Networks

TL;DR

This work tackles edge-LLM fine-tuning under resource limits, non-IID data, and privacy constraints. It presents ELSA, a hierarchical, split-learning framework that fuses SL and HFL with behavior-aware clustering, a tripartite LLM split, and semantic subspace orthogonal perturbation (SS-OP) with computational sketches to preserve privacy and reduce communication. Across eight NLP tasks, ELSA demonstrates faster convergence and higher final accuracy while delivering substantial communication savings, including up to about -fold throughput gains under suitable compression ratios. The approach offers a scalable, privacy-aware path for edge-adaptive LLM fine-tuning and can be extended to ultra-large models with adaptive splitting and compression strategies.

Abstract

Training large language models (LLMs) at the network edge faces fundamental challenges arising from device resource constraints, severe data heterogeneity, and heightened privacy risks. To address these, we propose ELSA (Efficient LLM-centric Split Aggregation), a novel framework that systematically integrates split learning (SL) and hierarchical federated learning (HFL) for distributed LLM fine-tuning over resource-constrained edge networks. ELSA introduces three key innovations. First, it employs a task-agnostic, behavior-aware client clustering mechanism that constructs semantic fingerprints using public probe inputs and symmetric KL divergence, further enhanced by prediction-consistency-based trust scoring and latency-aware edge assignment to jointly address data heterogeneity, client unreliability, and communication constraints. Second, it splits the LLM into three parts across clients and edge servers, with the cloud used only for adapter aggregation, enabling an effective balance between on-device computation cost and global convergence stability. Third, it incorporates a lightweight communication scheme based on computational sketches combined with semantic subspace orthogonal perturbation (SS-OP) to reduce communication overhead while mitigating privacy leakage during model exchanges. Experiments across diverse NLP tasks demonstrate that ELSA consistently outperforms state-of-the-art methods in terms of adaptability, convergence behavior, and robustness, establishing a scalable and privacy-aware solution for edge-side LLM fine-tuning under resource constraints.
Paper Structure (19 sections, 1 theorem, 35 equations, 6 figures, 4 tables)

This paper contains 19 sections, 1 theorem, 35 equations, 6 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Relate Work
  3. FL-driven Fine-Tuning of LLMs over Edge Networks
  4. Review of the Use of SL over Edge Networks
  5. Proposed Method
  6. Problem Formulation
  7. HFL with Tripartite Split Training
  8. Behavior-aware Hierarchical Clustering
  9. Model Splitting and Collaborative Training
  10. Layered Compression for Secure and Efficient Communications
  11. Communication Modeling
  12. Convergence Analysis of ELSA
  13. Experiments and Evaluations
  14. Experimental Setup
  15. Evaluation of Learning Efficiency and Model Performance
  16. ...and 4 more sections

Key Result

Theorem 1

Let $\eta = \frac{1}{\mathcal{L}\sqrt{G}}$, the sequence of global adapter parameters $\{\theta_g\}_{g=0}^{G-1}$ generated by ELSA satisfies: where $\widetilde{F}^* = \min_\theta \widetilde{F}(\theta)$ denotes the optimal value of the global objective, and please see proof in Appendix.

Figures (6)

  • Figure 1: A schematic of our proposed ELSA: clients and edge servers collaboratively fine-tuning LLMs via LoRA, with activations compressed during transmission; the cloud globally aggregates the LoRA parameters.
  • Figure 2: Client behavioral heterogeneity and clustering outcome in a 20-client network. Left: $20 \times 20$ pairwise KL divergence matrix $\mathcal{R}(n,n')$ across all clients. Right: final client-server association mapping.
  • Figure 3: Breakdown of ELSA: client trains Part 1, sends compressed activations to edge for Part 2, which then feeds into Part 3 on client. Gradients flow backward symmetrically ($\nabla \widetilde{\mathbf{H}}_{n}^{\mathsf{down}} \xrightarrow{} \text{Edge}$, $\nabla \widetilde{\mathbf{H}}_{n}^{\mathsf{up}} \xrightarrow{} \text{Client}$), completing one round.
  • Figure 4: Test performance comparison across TC datasets under two levels of data heterogeneity: (a)–(d) correspond to $\hat{\alpha}=$ 0.1, while (e)–(h) use $\hat{\alpha}=$ 0.2.
  • Figure 5: Evaluation performance and communication reduction of ELSA versus uncompressed schemes on various datasets (Indicator are Acc. (RTE, CB) and F1 (MultiRC, SQuAD), compression ratio $\rho$ = 4.2).
  • ...and 1 more figures

Theorems & Definitions (3)

  • Theorem 1
  • proof
  • Remark 1