Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FedLoRA-Optimizer: Federated LoRA Fine-Tuning with Global and Local Optimization in Heterogeneous Data Scenarios

Jianzhe Zhao, Hailin Zhu, Yu Zhang, Ziqi Chen, Guibing Guo

TL;DR

The paper tackles federated fine-tuning of large language models under heterogeneous data using LoRA. It reveals that LoRA’s A (direction) and B (magnitude) components have distinct sensitivities, with A capturing global knowledge and B capturing task-specific personalization. The authors propose FedLoRA-Optimizer, a global–local pipeline that separately optimizes A directions for global generalization and B magnitudes for personalization, implemented via a decomposition A = A_M A_D and B = B_M B_D and a combination of Federated Averaging and local fine-tuning. Empirical results on LLaMA2-7B and DeepSeek-7B across Databricks-Dolly-15k and Natural Instructions show modest but consistent gains in global and local performance, validating the approach and highlighting pipeline structure as beneficial. These findings suggest a practical path toward scalable, parameter-efficient federated fine-tuning in heterogeneous environments, with avenues for further optimization.

Abstract

Federated efficient fine-tuning has emerged as an approach that leverages distributed data and computational resources across nodes to address the challenges of large-scale fine-tuning and privacy preservation. The Low-Rank Adaptation (LoRA) enables efficient fine-tuning of large-scale pre-trained models by introducing trainable low-rank matrices into weight updates.However, in heterogeneous data scenarios, client drift weakens the generalization of the global model, and local models often fail to meet the personalized needs of individual clients.Moreover, existing federated LoRA efficient fine-tuning techniques overlook fine-grained analysis of the tuning matrices. To address this, we conducted preliminary experiments and found that different LoRA matrices exhibit different sensitivity to changes in the direction and magnitude of their vectors.We thus propose a fine-grained federated LoRA tuning method. By fine-tuning the more sensitive directional vectors in the A matrix, which encode shared knowledge, our method learns shared features more effectively across clients and enhances global generalization. Simultaneously, by fine-tuning the more sensitive magnitude vectors in the B matrix, which encode personalized knowledge, our method better captures personalized knowledge, enabling detailed adaptation to local data. The method uses a pipeline combining global and local optimizers. Global optimization further improves local models, achieving collaborative optimization between global and local levels. This improves both the generalization ability of the global model and the personalized adaptation of local models under heterogeneous data scenarios. Experiments on Databricks-Dolly-15k and Natural Instructions with LLaMA2-7B and Deepseek-7B confirm that our method improves global performance by 0.39% and local performance by 0.59%.

FedLoRA-Optimizer: Federated LoRA Fine-Tuning with Global and Local Optimization in Heterogeneous Data Scenarios

TL;DR

The paper tackles federated fine-tuning of large language models under heterogeneous data using LoRA. It reveals that LoRA’s A (direction) and B (magnitude) components have distinct sensitivities, with A capturing global knowledge and B capturing task-specific personalization. The authors propose FedLoRA-Optimizer, a global–local pipeline that separately optimizes A directions for global generalization and B magnitudes for personalization, implemented via a decomposition A = A_M A_D and B = B_M B_D and a combination of Federated Averaging and local fine-tuning. Empirical results on LLaMA2-7B and DeepSeek-7B across Databricks-Dolly-15k and Natural Instructions show modest but consistent gains in global and local performance, validating the approach and highlighting pipeline structure as beneficial. These findings suggest a practical path toward scalable, parameter-efficient federated fine-tuning in heterogeneous environments, with avenues for further optimization.

Abstract

Federated efficient fine-tuning has emerged as an approach that leverages distributed data and computational resources across nodes to address the challenges of large-scale fine-tuning and privacy preservation. The Low-Rank Adaptation (LoRA) enables efficient fine-tuning of large-scale pre-trained models by introducing trainable low-rank matrices into weight updates.However, in heterogeneous data scenarios, client drift weakens the generalization of the global model, and local models often fail to meet the personalized needs of individual clients.Moreover, existing federated LoRA efficient fine-tuning techniques overlook fine-grained analysis of the tuning matrices. To address this, we conducted preliminary experiments and found that different LoRA matrices exhibit different sensitivity to changes in the direction and magnitude of their vectors.We thus propose a fine-grained federated LoRA tuning method. By fine-tuning the more sensitive directional vectors in the A matrix, which encode shared knowledge, our method learns shared features more effectively across clients and enhances global generalization. Simultaneously, by fine-tuning the more sensitive magnitude vectors in the B matrix, which encode personalized knowledge, our method better captures personalized knowledge, enabling detailed adaptation to local data. The method uses a pipeline combining global and local optimizers. Global optimization further improves local models, achieving collaborative optimization between global and local levels. This improves both the generalization ability of the global model and the personalized adaptation of local models under heterogeneous data scenarios. Experiments on Databricks-Dolly-15k and Natural Instructions with LLaMA2-7B and Deepseek-7B confirm that our method improves global performance by 0.39% and local performance by 0.59%.

Paper Structure

This paper contains 13 sections, 12 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Motivation and Observation
  4. Proposed Methodology
  5. Outline
  6. Fine-Tuning Techniques for Global Model Optimization
  7. Efficient Fine-Tuning Techniques for Global and Local Fusion
  8. Experiment
  9. Experimental Settings
  10. Overall Performance
  11. Parameter Analysis
  12. Ablation Studies
  13. Conclusion

Figures (3)

  • Figure 1: Sensitivity of federated LoRA fine-tuning to magnitude changes and direction changes in A and B matrices; we conducted training using the Databricks-Dolly-15K dataset on the LLaMA2-7B model.
  • Figure 2: Architecture diagram of federated LoRA Fine-Tuning for global and local.
  • Figure 3: The comparison of accuracy across various tasks between the model that combines a global optimizer and a local optimizer in series and the model that uses only a local optimizer.In this study, we conducted experiments on the three aforementioned tasks using the DeepSeek-7B model on the Databricks-Dolly-15k dataset.