Efficient Federated Fine-Tuning of Large Language Models with Layer Dropout
Shilong Wang, Jianchun Liu, Hongli Xu, Jiaming Yan, Xianjun Gao
TL;DR
This paper tackles the heavy on-device cost of federated fine-tuning for large language models by introducing DropPEFT, which combines stochastic transformer layer dropout (STLD), an online dropout-rate configurator, and personalized transformer layer sharing (PTLS). STLD dynamically shortens network depth during local training, significantly reducing compute and memory while maintaining full model capacity for inference; the configurator optimizes dropout per device, and PTLS mitigates non-IID data effects by balancing shared and personalized layers. Empirical results show 1.3-6.3× faster convergence, 40-67% memory reduction, and notable gains in accuracy across multiple datasets and models, with substantial reductions in energy and network traffic. The approach offers a practical, privacy-friendly pathway to efficient federated fine-tuning of state-of-the-art LLMs on resource-constrained devices, with clear gains over existing PEFT-based methods.
Abstract
Fine-tuning plays a crucial role in enabling pre-trained LLMs to evolve from general language comprehension to task-specific expertise. To preserve user data privacy, federated fine-tuning is often employed and has emerged as the de facto paradigm. However, federated fine-tuning is prohibitively inefficient due to the tension between LLM complexity and the resource constraint of end devices, incurring unaffordable fine-tuning overhead. Existing literature primarily utilizes parameter-efficient fine-tuning techniques to mitigate communication costs, yet computational and memory burdens continue to pose significant challenges for developers. This work proposes DropPEFT, an innovative federated PEFT framework that employs a novel stochastic transformer layer dropout method, enabling devices to deactivate a considerable fraction of LLMs layers during training, thereby eliminating the associated computational load and memory footprint. In DropPEFT, a key challenge is the proper configuration of dropout ratios for layers, as overhead and training performance are highly sensitive to this setting. To address this challenge, we adaptively assign optimal dropout-ratio configurations to devices through an exploration-exploitation strategy, achieving efficient and effective fine-tuning. Extensive experiments show that DropPEFT can achieve a 1.3-6.3\times speedup in model convergence and a 40%-67% reduction in memory footprint compared to state-of-the-art methods.