LoRA-Pro: Are Low-Rank Adapters Properly Optimized?

TL;DR

LoRA-Pro addresses the gap between LoRA and full fine-tuning by revealing that LoRA optimization equates to applying a low-rank gradient in full fine-tuning. It then optimizes the gradients of the LoRA components A and B to minimize the distance to the true gradient, with a closed-form solution and practical algorithm. Extensive experiments across NLP, dialogue, reasoning, code, and vision tasks show that LoRA-Pro consistently improves LoRA performance and narrows the gap to full fine-tuning, often matching or surpassing it on average. The work also analyzes rank behavior, ablations on gradient adjustments, and shows favorable memory-time trade-offs, providing a scalable, trainable proxy to full fine-tuning.

Abstract

Low-rank adaptation, also known as LoRA, has emerged as a prominent method for parameter-efficient fine-tuning of foundation models. Despite its computational efficiency, LoRA still yields inferior performance compared to full fine-tuning. In this paper, we first uncover a fundamental connection between the optimization processes of LoRA and full fine-tuning: using LoRA for optimization is mathematically equivalent to full fine-tuning using a low-rank gradient for parameter updates. And this low-rank gradient can be expressed in terms of the gradients of the two low-rank matrices in LoRA. Leveraging this insight, we introduce LoRA-Pro, a method that enhances LoRA's performance by strategically adjusting the gradients of these low-rank matrices. This adjustment allows the low-rank gradient to more accurately approximate the full fine-tuning gradient, thereby narrowing the performance gap between LoRA and full fine-tuning. Furthermore, we theoretically derive the optimal solutions for adjusting the gradients of the low-rank matrices, applying them during fine-tuning in LoRA-Pro. We conduct extensive experiments across natural language understanding, dialogue generation, mathematical reasoning, code generation, and image classification tasks, demonstrating that LoRA-Pro substantially improves LoRA's performance, effectively narrowing the gap with full fine-tuning. Code is publicly available at https://github.com/mrflogs/LoRA-Pro.

Figures (4)

• Figure 1: Illustration of LoRA-Pro. LoRA hu2022lora reduces the trainable parameter by re-parameterizing the weight into the product of two low-rank matrices, i.e., $W = W_0 + sBA$. We have discovered a connection between the optimization processes of full fine-tuning and LoRA. Updating matrices $B$ and $A$ using gradients $g^B$ and $g^A$ is equivalent to updating weight $W$ using a virtual low-rank gradient $\tilde{g} = sBg^A + sg^BA$. Therefore, in LoRA-Pro, we aim to adjust gradients $g^B$ and $g^A$ to minimize the distance between the equivalent gradient $\tilde{g}$ and the full fine-tuning gradient $g$, thereby reducing their performance gap. In Theorem \ref{['thm:close_form']}, we provide the optimal update gradients, and in Appendix \ref{['sec:appendix_algorithm']}, we present the pseudo-code for the optimization algorithm.
• Figure 2: Visualization of matrix ranks of A and B during training, with ranks set to $8$ and $32$, respectively.
• Figure 3: Training loss curves of LoRA, LoRA-GA, LoRA-Pro, and Full Fine-tuning on WizardLM, MetaMathQA, and CodeFeedback.
• Figure 4: Visualization of the differences between the equivalent gradients of LoRA, LoRA-Pro, and the full-parameter gradients during training, i.e., $\|\tilde{g} - g\|_F$. The rows illustrate the differences across various modules, including Q, K, V, O, Up, Down, and Gate. The columns show the differences at different depths, categorized as shallow (1), medium (15), and deep layers (31).

Theorems & Definitions (15)

• Definition 1: Equivalent Gradient
• Theorem 2.1
• proof
• Theorem 2.2
• proof
• Theorem 2.3
• proof
• Lemma
• proof
• Theorem