GoRA: Gradient-driven Adaptive Low Rank Adaptation

TL;DR

GoRA addresses the memory-efficiency gap in fine-tuning large language models by jointly optimizing LoRA rank allocation and initialization using gradient information. By reinterpreting LoRA as gradient compressors, GoRA performs adaptive per-weight rank allocation and data-driven, nonmanipulative initialization via a pseudo-inverse compression of accumulated gradients, while preserving a parameter budget close to vanilla LoRA. Extensive NLP and vision experiments show GoRA consistently outperforms LoRA baselines and even rivals full fine-tuning under high-rank settings, with minimal overhead and strong compatibility with QLoRA (QGoRA). The approach offers practical, auto-tunable strategies and scalable applicability across modalities, enabling more effective, efficient adaptation of large pre-trained models in resource-constrained environments.

Abstract

Low-Rank Adaptation (LoRA) is a crucial method for efficiently fine-tuning large language models (LLMs), with its effectiveness influenced by two key factors: rank selection and weight initialization. While numerous LoRA variants have been proposed to improve performance by addressing one of these aspects, they often compromise usability or computational efficiency. In this paper, we analyze and identify the core limitations of existing approaches and propose a novel framework--GoRA (Gradient-driven Adaptive Low Rank Adaptation)--that simultaneously adapts both the rank and initialization strategy within a unified framework. GoRA leverages gradient information during training to dynamically assign optimal ranks and initialize low-rank adapter weights in an adaptive manner. To our knowledge, GoRA is the first method that not only addresses the limitations of prior approaches--which often focus on either rank selection or initialization in isolation--but also unifies both aspects within a single framework, enabling more effective and efficient adaptation. Extensive experiments across various architectures and modalities show that GoRA consistently outperforms existing LoRA-based methods while preserving the efficiency of vanilla LoRA. For example, when fine-tuning Llama3.1-8B-Base for mathematical reasoning, GoRA achieves a 5.13-point improvement over standard LoRA and even outperforms full fine-tuning by 2.05 points under high-rank settings. Code is available at: https://github.com/hhnqqq/MyTransformers.

Figures (3)

• Figure 1: Illustration of (a) LoRA; (b) LoRA variants utilizing adaptive rank masking strategies; (c) LoRA variants employing nonzero initialization strategies; and (d) GoRA, which introduces adaptively leveraging the weight $\mathbf{W}$'s gradient to allocate the rank of the low-rank adapter and initialize $\mathbf{B}$. $\mathbf{A}_0$ and $\mathbf{B}_0$ denote the initialized value of $\mathbf{A}$ and $\mathbf{B}$.
• Figure 2: The training loss curves of full fine-tuning, LoRA, GoRA, and $\text{GoRA}_{r_0=128}$ on Llama-3.1-8B-Base. GoRA demonstrates lower start loss and faster convergence speed.
• Figure 3: (a) Result rank distribution of fine-tuning Llama-3.1-8B-Base on the MetaMathQA-100K dataset using GoRA;(b) Difference values between GoRA and LoRA in directional updates of pre-trained weights after merging;(c) Difference values between GoRA and LoRA in magnitude updates of pre-trained weights after merging. Data points are presented for every two layers.