Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Federated Low-Rank Adaptation for Foundation Models: A Survey

Yiyuan Yang, Guodong Long, Qinghua Lu, Liming Zhu, Jing Jiang, Chengqi Zhang

TL;DR

FedLoRA addresses the privacy and efficiency challenges of fine-tuning foundation models by learning and communicating only low-rank LoRA adapters in a federated setting. The survey provides a structured taxonomy across distributed learning, heterogeneity, and efficiency, and surveys methods for server aggregation, initialization, personalization, and efficiency improvements, highlighting both practical benefits and theoretical gaps. It synthesizes techniques from discordant LoRA aggregation to full-size aggregation, rank clustering, and personalized LoRA, outlining when each approach is advantageous. The work identifies open questions, such as convergence guarantees under heterogeneity and standardized benchmarks, and points to promising directions like adaptive rank, hypernetwork-based personalization, and broader domain applications.

Abstract

Effectively leveraging private datasets remains a significant challenge in developing foundation models. Federated Learning (FL) has recently emerged as a collaborative framework that enables multiple users to fine-tune these models while mitigating data privacy risks. Meanwhile, Low-Rank Adaptation (LoRA) offers a resource-efficient alternative for fine-tuning foundation models by dramatically reducing the number of trainable parameters. This survey examines how LoRA has been integrated into federated fine-tuning for foundation models, an area we term FedLoRA, by focusing on three key challenges: distributed learning, heterogeneity, and efficiency. We further categorize existing work based on the specific methods used to address each challenge. Finally, we discuss open research questions and highlight promising directions for future investigation, outlining the next steps for advancing FedLoRA.

Federated Low-Rank Adaptation for Foundation Models: A Survey

TL;DR

FedLoRA addresses the privacy and efficiency challenges of fine-tuning foundation models by learning and communicating only low-rank LoRA adapters in a federated setting. The survey provides a structured taxonomy across distributed learning, heterogeneity, and efficiency, and surveys methods for server aggregation, initialization, personalization, and efficiency improvements, highlighting both practical benefits and theoretical gaps. It synthesizes techniques from discordant LoRA aggregation to full-size aggregation, rank clustering, and personalized LoRA, outlining when each approach is advantageous. The work identifies open questions, such as convergence guarantees under heterogeneity and standardized benchmarks, and points to promising directions like adaptive rank, hypernetwork-based personalization, and broader domain applications.

Abstract

Effectively leveraging private datasets remains a significant challenge in developing foundation models. Federated Learning (FL) has recently emerged as a collaborative framework that enables multiple users to fine-tune these models while mitigating data privacy risks. Meanwhile, Low-Rank Adaptation (LoRA) offers a resource-efficient alternative for fine-tuning foundation models by dramatically reducing the number of trainable parameters. This survey examines how LoRA has been integrated into federated fine-tuning for foundation models, an area we term FedLoRA, by focusing on three key challenges: distributed learning, heterogeneity, and efficiency. We further categorize existing work based on the specific methods used to address each challenge. Finally, we discuss open research questions and highlight promising directions for future investigation, outlining the next steps for advancing FedLoRA.
Paper Structure (38 sections, 16 equations, 4 figures, 1 table)

This paper contains 38 sections, 16 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Contributions.
  3. Preliminary
  4. Federated Learning
  5. Low-Rank Adaptation
  6. Variations of LoRA.
  7. Federated Low-Rank Adaptation
  8. Taxonomy.
  9. Distributed Learning
  10. Server Aggregation
  11. LoRA Aggregation Discordance.
  12. Single Low-Rank Matrix Aggregation.
  13. Full-size Matrix Aggregation.
  14. Corrective Mechanism.
  15. Rank Clustering Aggregation.
  16. ...and 23 more sections

Figures (4)

  • Figure 1: The overall framework of FedLoRA, where only the LoRA parameters ($A$ and $B$) are communicated for efficient learning.
  • Figure 2: Taxonomy of FedLoRA focusing on distributed learning, heterogeneity and efficiency with further classified subcategories.
  • Figure 3: Two types of sparse learning frameworks in FedLoRA.
  • Figure 4: Two types of split learning frameworks in FedLoRA.