FedLoDrop: Federated LoRA with Dropout for Generalized LLM Fine-tuning
Sijing Xie, Dingzhu Wen, Changsheng You, Qimei Chen, Mehdi Bennis, Kaibin Huang
TL;DR
FedLoDrop tackles the challenge of generalizable, cost-efficient fine-tuning of large language models in federated edge settings by injecting dropout into Low-Rank Adaptation (LoRA) adapters and coordinating updates across clients. The authors derive a pointwise hypothesis stability (PHS) based generalization bound, show how dropout controls the trade-off between underfitting and overfitting, and formulate a joint dropout-resource optimization problem. They propose a Branch-and-Bound (B&B) exact solver and a low-complexity Penalized-SCA (P-SCA) method to minimize the bound under latency, energy, and subcarrier constraints, and validate effectiveness via simulations on RoBERTa-large and LLaMA with notable reductions in communication and improved generalization. The work provides practical, edge-aware federated fine-tuning strategies with theoretical guarantees and scalable solutions for real-world deployment of PEFT methods in distributed environments.
Abstract
Fine-tuning (FT) large language models (LLMs) is crucial for adapting general-purpose models to specific tasks, enhancing accuracy and relevance with minimal resources. To further enhance generalization ability while reducing training costs, this paper proposes Federated LoRA with Dropout (FedLoDrop), a new framework that applies dropout to the rows and columns of the trainable matrix in Federated LoRA. A generalization error bound and convergence analysis under sparsity regularization are obtained, which elucidate the fundamental trade-off between underfitting and overfitting. The error bound reveals that a higher dropout rate increases model sparsity, thereby lowering the upper bound of pointwise hypothesis stability (PHS). While this reduces the gap between empirical and generalization errors, it also incurs a higher empirical error, which, together with the gap, determines the overall generalization error. On the other hand, though dropout reduces communication costs, deploying FedLoDrop at the network edge still faces challenges due to limited network resources. To address this issue, an optimization problem is formulated to minimize the upper bound of the generalization error, by jointly optimizing the dropout rate and resource allocation subject to the latency and per-device energy consumption constraints. To solve this problem, a branch-and-bound (B\&B)-based method is proposed to obtain its globally optimal solution. Moreover, to reduce the high computational complexity of the B\&B-based method, a penalized successive convex approximation (P-SCA)-based algorithm is proposed to efficiently obtain its high-quality suboptimal solution. Finally, numerical results demonstrate the effectiveness of the proposed approach in mitigating overfitting and improving the generalization capability.