Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Robust Federated Finetuning of Foundation Models via Alternating Minimization of LoRA

Shuangyi Chen, Yue Ju, Hardik Dalal, Zhongwen Zhu, Ashish Khisti

TL;DR

RoLoRA is introduced, a robust federated fine-tuning framework that utilizes an alternating minimization approach for LoRA, providing greater robustness against decreasing fine-tuning parameters and increasing data heterogeneity, and substantially enhances the robustness and effectiveness in multiple federated fine-tuning scenarios.

Abstract

Parameter-Efficient Fine-Tuning (PEFT) has risen as an innovative training strategy that updates only a select few model parameters, significantly lowering both computational and memory demands. PEFT also helps to decrease data transfer in federated learning settings, where communication depends on the size of updates. In this work, we explore the constraints of previous studies that integrate a well-known PEFT method named LoRA with federated fine-tuning, then introduce RoLoRA, a robust federated fine-tuning framework that utilizes an alternating minimization approach for LoRA, providing greater robustness against decreasing fine-tuning parameters and increasing data heterogeneity. Our results indicate that RoLoRA not only presents the communication benefits but also substantially enhances the robustness and effectiveness in multiple federated fine-tuning scenarios.

Robust Federated Finetuning of Foundation Models via Alternating Minimization of LoRA

TL;DR

RoLoRA is introduced, a robust federated fine-tuning framework that utilizes an alternating minimization approach for LoRA, providing greater robustness against decreasing fine-tuning parameters and increasing data heterogeneity, and substantially enhances the robustness and effectiveness in multiple federated fine-tuning scenarios.

Abstract

Parameter-Efficient Fine-Tuning (PEFT) has risen as an innovative training strategy that updates only a select few model parameters, significantly lowering both computational and memory demands. PEFT also helps to decrease data transfer in federated learning settings, where communication depends on the size of updates. In this work, we explore the constraints of previous studies that integrate a well-known PEFT method named LoRA with federated fine-tuning, then introduce RoLoRA, a robust federated fine-tuning framework that utilizes an alternating minimization approach for LoRA, providing greater robustness against decreasing fine-tuning parameters and increasing data heterogeneity. Our results indicate that RoLoRA not only presents the communication benefits but also substantially enhances the robustness and effectiveness in multiple federated fine-tuning scenarios.
Paper Structure (29 sections, 2 equations, 2 figures, 8 tables)

This paper contains 29 sections, 2 equations, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Low-Rank Adaptation: LoRA
  5. FedAVG of LoRA Introduces Interference
  6. FedRep: Common Representation via Alternating Minimization
  7. Our Framework
  8. Alternating Minimization and Corresponding Aggregation
  9. Computation and Communication Cost
  10. Experiments
  11. Baselines
  12. Model and Datasets.
  13. Implementation.
  14. Effect of Number of Finetuning Parameters
  15. Effect of Data Heterogeneity
  16. ...and 14 more sections

Figures (2)

  • Figure 1: Results with RoBERTa-Large models on GLUE of different budget of finetuning parameters. The accuracy is computed by averaging over different ranks $\{1,2,4,8\}$. The number of clients is 3.
  • Figure 2: Accuracies over rounds with RoBERTa-Large models on SST-2, QNLI, MNLI, and QQP. The total number of clients is 50.