Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DLoRA: Distributed Parameter-Efficient Fine-Tuning Solution for Large Language Model

Chao Gao, Sai Qian Zhang

TL;DR

DLoRA addresses privacy and scalability challenges in distributed fine-tuning of large language models by offloading PEFT workloads to edge devices while keeping user data on-device. It introduces the Kill and Revive (KR) mechanism to dynamically select a small, task-relevant subset of PEFT modules, drastically reducing edge compute and cloud–edge communication without sacrificing accuracy. Empirical results across multiple LLMs and eight tasks show roughly an 82% reduction in compute and 87.5% in communication, with competitive or improved accuracy compared to cloud-only baselines and several pruning baselines. The approach preserves privacy, enables scalable personalization, and remains effective when paired with Adapters, as well as under low-bit quantization, highlighting practical impact for real-world edge–cloud deployments.

Abstract

To enhance the performance of large language models (LLM) on downstream tasks, one solution is to fine-tune certain LLM parameters and make it better align with the characteristics of the training dataset. This process is commonly known as parameter-efficient fine-tuning (PEFT). Due to the scale of LLM, PEFT operations are usually executed in the public environment (e.g., cloud server). This necessitates the sharing of sensitive user data across public environments, thereby raising potential privacy concerns. To tackle these challenges, we propose a distributed PEFT framework called DLoRA. DLoRA enables scalable PEFT operations to be performed collaboratively between the cloud and user devices. Coupled with the proposed Kill and Revive algorithm, the evaluation results demonstrate that DLoRA can significantly reduce the computation and communication workload over the user devices while achieving superior accuracy and privacy protection.

DLoRA: Distributed Parameter-Efficient Fine-Tuning Solution for Large Language Model

TL;DR

DLoRA addresses privacy and scalability challenges in distributed fine-tuning of large language models by offloading PEFT workloads to edge devices while keeping user data on-device. It introduces the Kill and Revive (KR) mechanism to dynamically select a small, task-relevant subset of PEFT modules, drastically reducing edge compute and cloud–edge communication without sacrificing accuracy. Empirical results across multiple LLMs and eight tasks show roughly an 82% reduction in compute and 87.5% in communication, with competitive or improved accuracy compared to cloud-only baselines and several pruning baselines. The approach preserves privacy, enables scalable personalization, and remains effective when paired with Adapters, as well as under low-bit quantization, highlighting practical impact for real-world edge–cloud deployments.

Abstract

To enhance the performance of large language models (LLM) on downstream tasks, one solution is to fine-tune certain LLM parameters and make it better align with the characteristics of the training dataset. This process is commonly known as parameter-efficient fine-tuning (PEFT). Due to the scale of LLM, PEFT operations are usually executed in the public environment (e.g., cloud server). This necessitates the sharing of sensitive user data across public environments, thereby raising potential privacy concerns. To tackle these challenges, we propose a distributed PEFT framework called DLoRA. DLoRA enables scalable PEFT operations to be performed collaboratively between the cloud and user devices. Coupled with the proposed Kill and Revive algorithm, the evaluation results demonstrate that DLoRA can significantly reduce the computation and communication workload over the user devices while achieving superior accuracy and privacy protection.
Paper Structure (28 sections, 3 equations, 7 figures, 7 tables, 2 algorithms)

This paper contains 28 sections, 3 equations, 7 figures, 7 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. LLM Computation
  4. Parameter Efficient Fine-Tuning
  5. LLM Implementation
  6. Federated Learning
  7. Kill and Revive Mechanism
  8. Computation Pattern for DLoRA
  9. Early Kill Mechanism
  10. Parameter Revival Mechanism
  11. Evaluation
  12. Experiment Setup
  13. Datasets and models:
  14. PEFT Settings:
  15. DLoRA System Configuration:
  16. ...and 13 more sections

Figures (7)

  • Figure 1: (a) Cloud-only solution. (b) Edge-only solution. (c) DLoRA scheme. The frozen and learnable parameters are shown in orange and red, respectively.
  • Figure 2: (a) LLaMA architecture. (b) LLaMA auto-regressive pattern. (c) LoRA operation. All the learnable components are highlighted in red, while the frozen components are highlighted in grey. LoRA is applied on all the query, key, and value blocks, we only show one of them for illustration simplicity.
  • Figure 3: $l_{2}$-norm variation of selected PEFT modules across training iterations over multiple downstream tasks including (a) Arc-Challenge, (b) Social-QA.
  • Figure 4: DLoRA computation pattern for one iteration.
  • Figure 5: Computation costs of KR and FT over different LLMs, the measurement denotes the computation cost in TFLOPs. The communication costs for KR and FT are measured in Gigabytes (GB).
  • ...and 2 more figures