Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On the Convergence Rate of LoRA Gradient Descent

Siqiao Mu, Diego Klabjan

TL;DR

The paper establishes the first non-asymptotic convergence rate for the original LoRA gradient descent without requiring bounded adapter norms or Lipschitz smoothness of the reparametrized loss. It achieves this by reformulating the problem as optimization over $V$ with $VV^T$ encoding $BA$, proving a Lipschitz-like descent lemma, and designing a carefully controlled step size. The main result shows a convergence rate of $O(1/\log T)$ to a stationary point, improving to $O(1/T)$ if the adapter norms are bounded, and it highlights that the convergence behavior is largely independent of the LoRA rank and can diverge from full-rank fine-tuning. The analysis connects to the Burer-Monteiro framework and clarifies how step-size and parameter growth influence LoRA dynamics, informing theoretical understanding and practical deployment of LoRA in parameter-efficient fine-tuning.

Abstract

The low-rank adaptation (LoRA) algorithm for fine-tuning large models has grown popular in recent years due to its remarkable performance and low computational requirements. LoRA trains two ``adapter" matrices that form a low-rank representation of the model parameters, thereby massively reducing the number of parameters that need to be updated at every step. Although LoRA is simple, its convergence is poorly understood due to the lack of Lipschitz smoothness, a key condition for classic convergence analyses. As a result, current theoretical results only consider asymptotic behavior or assume strong boundedness conditions which artificially enforce Lipschitz smoothness. In this work, we provide for the first time a non-asymptotic convergence analysis of the \textit{original LoRA gradient descent} algorithm, which reflects widespread practice, without such assumptions. Our work relies on three key steps: i) reformulating the problem in terms of the outer product of the stacked adapter matrices, ii) a modified descent lemma for the ``Lipschitz-like" reparametrized function, and iii) controlling the step size. With this approach, we prove that LoRA gradient descent converges to a stationary point at rate $O(\frac{1}{\log T})$, where $T$ is the number of iterations.

On the Convergence Rate of LoRA Gradient Descent

TL;DR

The paper establishes the first non-asymptotic convergence rate for the original LoRA gradient descent without requiring bounded adapter norms or Lipschitz smoothness of the reparametrized loss. It achieves this by reformulating the problem as optimization over with encoding , proving a Lipschitz-like descent lemma, and designing a carefully controlled step size. The main result shows a convergence rate of to a stationary point, improving to if the adapter norms are bounded, and it highlights that the convergence behavior is largely independent of the LoRA rank and can diverge from full-rank fine-tuning. The analysis connects to the Burer-Monteiro framework and clarifies how step-size and parameter growth influence LoRA dynamics, informing theoretical understanding and practical deployment of LoRA in parameter-efficient fine-tuning.

Abstract

The low-rank adaptation (LoRA) algorithm for fine-tuning large models has grown popular in recent years due to its remarkable performance and low computational requirements. LoRA trains two ``adapter" matrices that form a low-rank representation of the model parameters, thereby massively reducing the number of parameters that need to be updated at every step. Although LoRA is simple, its convergence is poorly understood due to the lack of Lipschitz smoothness, a key condition for classic convergence analyses. As a result, current theoretical results only consider asymptotic behavior or assume strong boundedness conditions which artificially enforce Lipschitz smoothness. In this work, we provide for the first time a non-asymptotic convergence analysis of the \textit{original LoRA gradient descent} algorithm, which reflects widespread practice, without such assumptions. Our work relies on three key steps: i) reformulating the problem in terms of the outer product of the stacked adapter matrices, ii) a modified descent lemma for the ``Lipschitz-like" reparametrized function, and iii) controlling the step size. With this approach, we prove that LoRA gradient descent converges to a stationary point at rate , where is the number of iterations.

Paper Structure

This paper contains 14 sections, 6 theorems, 56 equations.

Table of Contents

  1. Introduction
  2. Related Work
  3. Convergence Analysis
  4. Preliminaries
  5. Results
  6. Step 1: Restructure problem into $V V^T$ form.
  7. Step 2: "Lipschitz-like" descent lemma.
  8. Step 3: Control step size to obtain convergence.
  9. Discussion
  10. Proofs
  11. Helper Lemmas
  12. Proof of Lemma \ref{['lemma:descent_lemma']}
  13. Proof of Lemma \ref{['lemma:onestep']}
  14. Proof of Theorem \ref{['thm:moneymaker']}

Key Result

Lemma 1

For $\mathcal{J}(V)$ defined in (eq:h), we have for all $V_1, V_2 \in \mathbb{R}^{(m+n) \times r}$,

Theorems & Definitions (11)

  • Lemma 1
  • Lemma 2
  • Theorem 3
  • Lemma 4
  • Lemma 5
  • proof
  • Lemma 6
  • proof
  • proof
  • proof
  • ...and 1 more