SplitLLM: Hierarchical Split Learning for Large Language Model over Wireless Network

TL;DR

The paper addresses privacy-preserving fine-tuning of LLMs over wireless networks under memory and communication constraints. It proposes SplitLLM, a cloud-edge-user hierarchical split learning framework that partitions the pre-trained model and LoRA adapters into three parts and trains only adapters, with forward passes conducted from user to cloud and aggregation performed by the cloud via FedAvg. Key contributions include a three-way model partition, a training protocol enabling parallel edge-user updates, and empirical evidence showing memory peak reductions up to 74% while maintaining comparable accuracy across IID and non-IID settings on CIFAR100 and MRPC. This approach enables scalable, personalized LLM fine-tuning in bandwidth-limited and memory-constrained edge environments, reducing server-side memory pressure while preserving learning performance.

Abstract

Fine-tuning a large language model (LLM) using the local data of edge users can enable personalized services and applications. For privacy protection, the prevalent solution adopts distributed learning for fine-tuning and integrates low-rank adaptation (LoRA) to reduce users' computational load. However, as the number of users increases, numerous users simultaneously communicate with the server, and multiple server-side models concurrently execute on the server, leading to significant communication congestion and memory pressure. In this paper, we propose a split learning (SL) scheme for fine-tuning LLM in wireless networks, which involves one cloud server, a small number of edge servers, and multiple users. Specifically, the pre-trained model and LoRA adapters are divided into three parts and deployed across the cloud, edge, and user sides. The training process follows the sequence of user, edge, and cloud, with forward and backward propagation achieved by transmitting activation and gradient. In each round, all edge servers and an equivalent number of users train in parallel, and only the LoRA adapters are updated. At the end of each round, all edge-side and user-side LoRA adapters are uploaded to the cloud for aggregation. Extensive simulation demonstrates that the proposed scheme can reduce peak memory usage up to 74% compared to the state-of-the-art benchmarks.

Figures (2)

• Figure 1: (a) In the SplitLLM scheme, edge servers and corresponding users are trained parallelly$\mathrm{;}$ (b) in the LoRA-based learning system, only the LoRA adapter is updated, and the pre-trained model is divided into three components.
• Figure 2: Performance on various neural architectures and datasets.