Least but not Last: Fine-tuning Intermediate Principal Components for Better Performance-Forgetting Trade-Offs

TL;DR

The paper addresses the performance-forgetting trade-off in Low-Rank Adaptation (LoRA) for large pre-trained models by analyzing how fine-tuning principal components affects knowledge retention. It introduces a unified, SVD-based LoRA framework that can select starting component index $s$ and rank $r$, generalizing PiSSA and MiLoRA to intermediate components, and demonstrates that tuning these intermediate components yields superior accuracy-forgetting trade-offs, especially under higher learning rates. Through parameter-space and feature-space analyses, it explains why extreme components are more prone to forgetting and shows a consistent U-shaped forgetting pattern across principal components. Empirically, the approach improves task performance and reduces forgetting across vision and NLP benchmarks, indicating practical benefits for continual learning and robust PEFT design.

Abstract

Low-Rank Adaptation (LoRA) methods have emerged as crucial techniques for adapting large pre-trained models to downstream tasks under computational and memory constraints. However, they face a fundamental challenge in balancing task-specific performance gains against catastrophic forgetting of pre-trained knowledge, where existing methods provide inconsistent recommendations. This paper presents a comprehensive analysis of the performance-forgetting trade-offs inherent in low-rank adaptation using principal components as initialization. Our investigation reveals that fine-tuning intermediate components leads to better balance and show more robustness to high learning rates than first (PiSSA) and last (MiLoRA) components in existing work. Building on these findings, we provide a practical approach for initialization of LoRA that offers superior trade-offs. We demonstrate in a thorough empirical study on a variety of computer vision and NLP tasks that our approach improves accuracy and reduces forgetting, also in continual learning scenarios.

Figures (18)

• Figure 1: Accuracy (left) and forgetting (right) when fine-tuning principal components on ImageNet1k pre-trained ViT-B to Caltech101. Forgetting shows a U-shape with most information lost at the extremes where existing methods PiSSA use the main, and MiLoRA the least components, respectively.
• Figure 2: PiSSA, MiLoRA and our proposed approach.
• Figure 3: (ImageNet1k $\rightarrow$ Caltech101) Changes to the diagonal in parameter space, $\operatorname{diag}(\Delta \Sigma_W)$, see Eq. \ref{['eq:delta_diag']}. We show the element-wise norm.
• Figure 6: (ImageNet1k $\rightarrow$ Caltech101) Changes to the diagonal in feature space, $\operatorname{diag}(\Delta \Sigma_Y)$, see Eq. \ref{['eq:delta_sigma_y']}. We show the element-wise norm.
• Figure 9: (ImageNet1k $\rightarrow$ Caltech101) Results of fine-tuning an ImageNet1k pre-trained ViT-Base to Caltech101, using different principal component slices with starting points $s$ (horizontal axis) and rank 32. From left to right, accuracy of Caltech101, forgetting of ImageNet1k, and sum of accuracies of Caltech101 and ImageNet1k at the end of fine-tuning.