Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Low-Rank Adaptation for Foundation Models: A Comprehensive Review

Menglin Yang, Jialin Chen, Jinkai Tao, Yifei Zhang, Jiahong Liu, Jiasheng Zhang, Qiyao Ma, Harshit Verma, Regina Zhang, Min Zhou, Irwin King, Rex Ying

TL;DR

This survey addresses the problem of efficiently adapting foundation models to downstream tasks. It surveys Low-Rank Adaptation (LoRA) as a parameter-efficient fine-tuning method and extends its analysis beyond LLMs to general foundation models. It presents a structured breakdown of technical foundations, frontiers, and applications, offering a taxonomy of rank-based and decomposition-based strategies, training enhancements, and theoretical insights. The work highlights how LoRA enables scalable, modular, and robust adaptation with minimal additional parameters, making it highly relevant for practitioners deploying large models in diverse domains.

Abstract

The rapid advancement of foundation modelslarge-scale neural networks trained on diverse, extensive datasetshas revolutionized artificial intelligence, enabling unprecedented advancements across domains such as natural language processing, computer vision, and scientific discovery. However, the substantial parameter count of these models, often reaching billions or trillions, poses significant challenges in adapting them to specific downstream tasks. Low-Rank Adaptation (LoRA) has emerged as a highly promising approach for mitigating these challenges, offering a parameter-efficient mechanism to fine-tune foundation models with minimal computational overhead. This survey provides the first comprehensive review of LoRA techniques beyond large Language Models to general foundation models, including recent techniques foundations, emerging frontiers and applications of low-rank adaptation across multiple domains. Finally, this survey discusses key challenges and future research directions in theoretical understanding, scalability, and robustness. This survey serves as a valuable resource for researchers and practitioners working with efficient foundation model adaptation.

Low-Rank Adaptation for Foundation Models: A Comprehensive Review

TL;DR

This survey addresses the problem of efficiently adapting foundation models to downstream tasks. It surveys Low-Rank Adaptation (LoRA) as a parameter-efficient fine-tuning method and extends its analysis beyond LLMs to general foundation models. It presents a structured breakdown of technical foundations, frontiers, and applications, offering a taxonomy of rank-based and decomposition-based strategies, training enhancements, and theoretical insights. The work highlights how LoRA enables scalable, modular, and robust adaptation with minimal additional parameters, making it highly relevant for practitioners deploying large models in diverse domains.

Abstract

The rapid advancement of foundation modelslarge-scale neural networks trained on diverse, extensive datasetshas revolutionized artificial intelligence, enabling unprecedented advancements across domains such as natural language processing, computer vision, and scientific discovery. However, the substantial parameter count of these models, often reaching billions or trillions, poses significant challenges in adapting them to specific downstream tasks. Low-Rank Adaptation (LoRA) has emerged as a highly promising approach for mitigating these challenges, offering a parameter-efficient mechanism to fine-tune foundation models with minimal computational overhead. This survey provides the first comprehensive review of LoRA techniques beyond large Language Models to general foundation models, including recent techniques foundations, emerging frontiers and applications of low-rank adaptation across multiple domains. Finally, this survey discusses key challenges and future research directions in theoretical understanding, scalability, and robustness. This survey serves as a valuable resource for researchers and practitioners working with efficient foundation model adaptation.
Paper Structure (40 sections, 15 equations, 6 figures, 6 tables)

This paper contains 40 sections, 15 equations, 6 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Basics
  3. Foundations
  4. Parameter Efficiency Enhancement
  5. Parameter Decomposition
  6. Parameter Pruning
  7. Parameter Freezing and Sharing
  8. Parameter Quantization
  9. Rank Adaptation
  10. Rank Refinement
  11. Rank Augmentation
  12. Training Process Improvements
  13. Theoretical Foundations
  14. Frontiers
  15. Advanced Architecture
  16. ...and 25 more sections

Figures (6)

  • Figure 1: LoRA with foundation models in diverse domains.
  • Figure 2: Overview of the LoRA landscape for foundation models, highlighting the core technical components, representative enhancements, and downstream applications covered in this survey.
  • Figure 3: Structure of LoRA for Foundation Models.
  • Figure 4: Illustration of parameter efficiency enhancement methods: decomposition, pruning, freezing and sharing, and quantization
  • Figure 5: Illustration of rank refinement and augmentation methods.
  • ...and 1 more figures