HSplitLoRA: A Heterogeneous Split Parameter-Efficient Fine-Tuning Framework for Large Language Models

TL;DR

HSplitLoRA tackles the challenge of privately fine-tuning large language models across heterogeneous edge devices by fusing split learning with LoRA-based PEFT. It introduces three innovations: an important weight identification strategy using RNGWP and a dynamic historical/current importance blend, an adaptive rank and model-splitting configuration that respects device budgets, and a noise-free adapter aggregation scheme based on concatenation of low-rank factors. Together, these components enable efficient, scalable, and robust LLM fine-tuning under resource constraints and device unavailability, achieving superior convergence speed and accuracy compared to strong baselines. The framework is validated on LLaMA-2-7B and GPT-2-L with realistic hardware, demonstrating practical impact for privacy-preserving, distributed LLM customization in heterogeneous environments.

Abstract

Recently, large language models (LLMs) have achieved remarkable breakthroughs, revolutionizing the natural language processing domain and beyond. Due to immense parameter sizes, fine-tuning these models with private data for diverse downstream tasks has become mainstream. Though federated learning (FL) offers a promising solution for fine-tuning LLMs without sharing raw data, substantial computing costs hinder its democratization. Moreover, in real-world scenarios, private client devices often possess heterogeneous computing resources, further complicating LLM fine-tuning. To combat these challenges, we propose HSplitLoRA, a heterogeneous parameter-efficient fine-tuning (PEFT) framework built on split learning (SL) and low-rank adaptation (LoRA) fine-tuning, for efficiently fine-tuning LLMs on heterogeneous client devices. HSplitLoRA first identifies important weights based on their contributions to LLM training. It then dynamically configures the decomposition ranks of LoRA adapters for selected weights and determines the model split point according to varying computing budgets of client devices. Finally, a noise-free adapter aggregation mechanism is devised to support heterogeneous adapter aggregation without introducing noise. Extensive experiments demonstrate that HSplitLoRA outperforms state-of-the-art benchmarks in training accuracy and convergence speed.

Figures (22)

• Figure 1: A scenario of LLM SL via PEFT with client devices.
• Figure 2: An illustration of Adapter houlsby2019parameter and LoRA hu2021lora fine-tuning methods, where ${{\bf{W}}_q}$, ${{\bf{W}}_k}$, ${{\bf{W}}_v}$, and ${{\bf{W}}_o}$ are trainable weights lin2023pushing.
• Figure 3: The normalized computing and communication overheads of LoRA and FT on LLaMA-2-7B, where $r$ is the decomposition rank of LoRA adapter.
• Figure 4: The training performance with different trainable weights on LLaMA-2-7B, where the trainable weights combinations corresponding to quantities 1, 2, 3, and 4 are $\left\{ {{{\bf{W}}_q}} \right\}$, $\left\{ {{{\bf{W}}_q},{{\bf{W}}_k}} \right\}$, $\left\{ {{{\bf{W}}_q},{{\bf{W}}_k},{{\bf{W}}_v}} \right\}$, and $\left\{ {{{\bf{W}}_q},{{\bf{W}}_k},{{\bf{W}}_v},{{\bf{W}}_o}} \right\}$, respectively.
• Figure 5: The PPL at the $400$-th and $500$-th training rounds and the STDs of the PPL across 100 rounds on LLaMA-2-7B.