LoRA vs Full Fine-tuning: An Illusion of Equivalence

TL;DR

LoRA and full fine-tuning update different parts of pre-trained weight spaces; spectral analysis reveals LoRA introduces intruder dimensions—new high-ranking singular vectors misaligned with pre-trained components—that correlate with forgetting. The study demonstrates a causal link by scaling intruder singular values, showing that reducing their magnitude can greatly lower forgetting with minimal impact on downstream task performance, though accumulation of intruder dimensions in continual learning can worsen performance compared with full fine-tuning. These findings challenge the view of strict equivalence between LoRA and full fine-tuning and highlight practical mitigation strategies and diagnostic tools for overfitting to fine-tuning tasks. Overall, the work emphasizes spectral structure as a key lens for understanding and improving parameter-efficient fine-tuning.

Abstract

Fine-tuning is a crucial paradigm for adapting pre-trained large language models to downstream tasks. Recently, methods like Low-Rank Adaptation (LoRA) have been shown to effectively fine-tune LLMs with an extreme reduction in trainable parameters. But, \emph{are their learned solutions really equivalent?} We study how LoRA and full-finetuning change pre-trained models by analyzing the model's weight matrices through the lens of their spectral properties. We find that LoRA and full fine-tuning yield weight matrices whose singular value decompositions exhibit very different structure: weight matrices trained with LoRA have new, high-ranking singular vectors, which we call \emph{intruder dimensions}, while those trained with full fine-tuning do not. Further, we extend the finding that LoRA forgets less than full fine-tuning and find its forgetting is vastly localized to the intruder dimension -- by causally intervening on the intruder dimensions by changing their associated singular values post-fine-tuning, we show that they cause forgetting. Moreover, scaling them down significantly improves modeling of the pre-training distribution with a minimal drop in downstream task performance. Given this, we should expect accumulating intruder dimensions to be harmful and lead to more forgetting. This will be amplified during continual learning because of sequentially fine-tuning, and we show that LoRA models do accumulate intruder dimensions here tend to perform worse in this setting, emphasizing the practicality of our findings.

Figures (24)

• Figure 1: LoRA and full fine-tuning update the parameter space differently. Similarity matricies of pre- and post-fine-tuning singular vectors for LLaMA2-7B that characterize the spectral differences introduced during fine-tuning. Full fine-tuning retains most of the pre-training structure, while LoRA has a diagonal shift. Color shows cosine similarity.
• Figure 2: Characterizing structural differences between solutions learnt by LoRA & Full Fine-tuning.a) We measure the changes to the SVD of the pre-trained weights made during fine-tuning. We observe intruder dimensions introduced by LoRA in top ranking singular vectors but not by full fine-tuning. b) Comparing a matrix fine-tuned with full fine-tuning or LoRA. c) The intruder dimension shows near-zero absolute cosine similarity with all pre-trained singular vectors, in contrast to other singular vectors of the finetuned matrix.
• Figure 3: LoRA and full fine-tuning learn distinct structural solutions. LoRA introduces intruder dimensions (represented by outlined columns).
• Figure 4: LoRA has intruder dimensions, whereas full fine-tuning does not. Here, we set $k=10$ and measure the impact of $\epsilon$ on the number of intruder dimensions measured. LoRA introduces many intruder dimensions in the top 10 ranked singular vectors, while full fine-tuning does not. Numbers are reported are the sums across the entire model.
• Figure 5: Evolution of an intruder dimension across training steps.(Left) Intruder dimensions, and their rank, in a LoRA fine-tuned weight matrix during fine-tuning. (Middle) Their associated singular values, which shows that the singular value associated with the intruder dimension increases. (Right) Test accuracy across training steps.

Theorems & Definitions (1)

• Definition 3.1