$α$-LoRA: Effective Fine-Tuning via Base Model Rescaling

TL;DR

This work introduces $\alpha$-LoRA, a reparameterization that scales frozen base weights by a row-wise vector $\bm{\alpha}$ to create a tunable contribution from the source task during transfer learning. Using Random Matrix Theory, the authors prove the existence of a nontrivial optimal $\alpha^*$ (typically not equal to 1) and derive its closed-form expression, along with an algorithm to estimate it in complex settings such as LLM fine-tuning. They provide theoretical guarantees under high-dimensional growth conditions and validate the approach with both linear and large-scale language-model experiments, showing improved generalization over standard LoRA with manageable overhead. The work offers a principled mechanism to balance source and target information in fine-tuning, with practical implications for more effective adaptation of large pre-trained models. Overall, $\alpha$-LoRA advances transfer learning by introducing a scalable, theory-backed means to weight base-model contributions during adaptation.

Abstract

Fine-tuning has proven to be highly effective in adapting pre-trained models to perform better on new desired tasks with minimal data samples. Among the most widely used approaches are reparameterization methods, which update a target module by augmenting its frozen weight matrix with an additional trainable weight matrix. The most prominent example is Low Rank Adaption (LoRA), which gained significant attention in recent years. In this paper, we introduce a new class of reparameterization methods for transfer learning, designed to enhance the generalization ability of fine-tuned models. We establish the effectiveness of our approach in a high-dimensional binary classification setting using tools from Random Matrix Theory, and further validate our theoretical findings through more realistic experiments, such as fine-tuning LLMs.

Figures (9)

• Figure 1: Theoretical Test Accuracy variation with $\alpha$ for $N = 5000$, $n = 40$, $p = 1000$, and the theoretical model is modified to take $\beta$ in $(0,1)$: ${\bm{\mu}}_\beta = \beta {\bm{\mu}} + \sqrt{1 - \beta^2} {\bm{\mu}}^\perp$, where $\Vert {\bm{\mu}} \Vert = \Vert {\bm{\mu}}^{\perp} \Vert = 0.8$. Finally the regularization parameters are: $\tilde{\gamma} = 2$ and $\gamma = 10^{-1}$.
• Figure 2: Variations of the optimal parameter $\alpha^\star$ with respect to the alignment between the source ${\bm{\mu}}$ and target ${\bm{\mu}}_\beta$ dataset means. These latter were chosen of norm $1$, $N = 2000$, $n = 200$ and $\gamma = \tilde{\gamma} = 1$.
• Figure 3: Test accuracy variation with $\alpha$ for different transfer learning schemes from the Amazon Review dataset blitzer2007biographies. The considered parameters here are: $N = 2000$, $n = 40$, $p = 400$, $\gamma = 10^{-1}$ and $\tilde{\gamma} = 2$.
• Figure 4: Test accuracy evolution of roberta-base finetuned on MNLI and RTE for a single fixed seed (seed $5$ for MNLI and seed $123$ for RTE).
• Figure 5: Test accuracy of roberta-base finetuned on RTE for different values of $\bm{\alpha}$ in the neighborhood of the obtained $\bm{\alpha^\star}$. The values of the parameters $\bm \alpha^\star$ in this experiment range between $0.85$ and $1.14$.

Theorems & Definitions (25)

• Remark 3.1: About the interpretability of our fine-tuned classifier
• Remark 3.2: About the regularization parameter $\gamma$
• Definition 3.3: Resolvent
• Definition 3.4: Deterministic equivalent hachem2007deterministic
• Lemma 3.5: Deterministic equivalent of ${\mathbf{Q}}$ and ${\mathbf{R}}$
• Theorem 4.2: Gaussianity of the fine-tuned Ridge model
• Proposition 4.3: Asymptotic test accuracy of ${\bm{w}}_\alpha$
• Theorem 4.4: Optimal $\alpha$
• Lemma A.1: Resolvent identity
• Lemma A.2: Sherman-Morisson