FDLoRA: Personalized Federated Learning of Large Language Model via Dual LoRA Tuning

TL;DR

FDLoRA tackles personalized LLM customization under federated learning by deploying two LoRA modules per client to separately capture personalized and global knowledge, while an AdaFusion mechanism adaptively merges them. Only the global LoRA is communicated to the server, reducing communication and computation costs, and the base LLM remains frozen to preserve broad knowledge. Empirical results on log‑analysis and medical‑diagnosis tasks show FDLoRA achieving state‑of‑the‑art performance and robust behavior across varying client counts and non‑IID degrees, with favorable trade‑offs between communication and computation. By integrating dual LoRA, variant PFL, and gradient‑free fusion, FDLoRA provides a practical and scalable approach to FL for LLMs with limited labeled data per client.

Abstract

Large language models (LLMs) have emerged as important components across various fields, yet their training requires substantial computation resources and abundant labeled data. It poses a challenge to robustly training LLMs for individual users (clients). To tackle this challenge, the intuitive idea is to introduce federated learning (FL), which can collaboratively train models on distributed private data. However, existing methods suffer from the challenges of data heterogeneity, system heterogeneity, and model size, resulting in suboptimal performance and high costs. In this work, we proposed a variant of personalized federated learning (PFL) framework, namely FDLoRA, which allows the client to be a single device or a cluster and adopts low-rank adaptation (LoRA) tuning. FDLoRA sets dual LoRA modules on each client to capture personalized and global knowledge, respectively, and only the global LoRA module uploads parameters to the central server to aggregate cross-client knowledge. Finally, an adaptive fusion approach is employed to combine the parameters of the dual LoRAs. This enables FDLoRA to make effective use of private data distributed across different clients, thereby improving performance on the client without incurring high communication and computing costs. We conducted extensive experiments in two practice scenarios. The results demonstrate that FDLoRA outperforms six baselines in terms of performance, stability, robustness, computation cost, and communication cost.

Figures (10)

• Figure 1: The illustration of both federated learning and personalized federated learning. Personalized federated learning aims to find a personalized model for each client.
• Figure 2: Comparison of principle between regular fine-tuning (left) and LoRA (right).
• Figure 3: Framework overview of FDLoRA.
• Figure 4: Trainable Parameters vs Non-Trainable Parameters.
• Figure 5: The average accuracy varies with the communication round for different settings of the number of clients $N={3, 5, 10}$ and InnerOpt step $K=3$.