FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decomposition

TL;DR

FeDeRA addresses the privacy and efficiency challenges of fine-tuning large language models in federated settings by extending LoRA with a singular-value decomposition–based initialization of the adapters. This initialization stabilizes weight updates across non-IID clients, enabling FeDeRA to achieve comparable or superior task performance to full-parameter fine-tuning while using only about 1% of trainable parameters and dramatically reducing training time. Across six datasets and three NLP tasks, FeDeRA consistently outperforms other PEFT methods under federated non-IID conditions and shows strong robustness to data heterogeneity. The work demonstrates practical impact by enabling high-performing, communication-efficient, privacy-preserving FL for transformer-based models in realistic, heterogeneous environments.

Abstract

Despite their exceptional performance on various tasks after fine-tuning, pre-trained language models (PLMs) face significant challenges due to growing privacy concerns with data in centralized training methods. We consider federated learning (FL) to fine-tune PLMs in this paper. However, the substantial number of parameters in PLMs poses significant difficulties for client devices with limited communication and computational resources. One promising solution is to exploit parameter-efficient fine-tuning (PEFT) into FL, which trains a much smaller set of parameters than full parameter fine-tuning (FFT). Although remarkably improving training efficiency, PEFT methods may lead to degraded performance especially when data across different clients are non i.i.d, as revealed by experimental results. To overcome this, we propose FeDeRA, which extends and improves a widely used PEFT method, i.e., low-rank adaption (LoRA). FeDeRA follows LoRA by decomposing the weight matrices of the PLMs into low-rank matrices, which allows for more efficient computation and parameter updates during fine-tuning. Different from LoRA which simply initializes these low-rank matrices by random sampling or zeros, the proposed FeDeRA initializes these matrices by the results of performing singular value decomposition (SVD) on the pre-trained weight matrices. Extensive experiments across various tasks and datasets show that FeDeRA outperforms the considered PEFT baselines and is comparable to or even surpasses FFT method within the FL setting in terms of task performance. Moreover, FeDeRA requires only 1% trainable paramentes compared to FFT, significantly reducing training time costs by more than 90% to achieve the same task performance level. The experimental results also highlight the robustness of FeDeRA against data heterogeneity, as it maintains stable task performance even as data heterogeneity increases.

Figures (9)

• Figure 1: The performance of adopting PEFT methods within an FL setting is evaluated at varying levels of data heterogeneity using DistilBERT fine-tuned on the 20Newsgroup dataset. Heterogeneous data is generated based on a Dirichlet distribution, where the hyperparameter $\alpha$ determines the degree of data heterogeneity—a lower $\alpha$ value indicates higher data heterogeneitylin2022fednlp.
• Figure 2: Illustration of LoRA and FeDeRA.
• Figure 3: Magnitude variation in consecutive global weight updates by FeDeRA and FedLR fine-tuning DistilBERT over 200 Communication Rounds on the 20 Newsgroups Dataset.
• Figure 4: Direction variation in consecutive global weight updates by FeDeRA and FedLR fine-tuning DistilBERT over 200 communication rounds on the 20Newsgroups dataset.
• Figure 5: Performance comparison on test accuracy with regards to different levels of data heterogeneity.