Parameter Efficient Fine-tuning via Explained Variance Adaptation

TL;DR

EVA introduces a variance-driven initialization for LoRA that provably maximizes the initial gradient signal by aligning the adapter update directions with the principal activation variances via incremental SVD on minibatch activations. It further redistributes the rank budget adaptively across weight matrices to concentrate capacity where activation variance is largest, controlled by a parameter ρ. The method yields faster convergence and higher average performance across language, vision, and RL tasks while reducing trainable parameters, achieving Pareto-dominant outcomes over existing LoRA initializations. The approach is grounded in gradient amplification and NTK perspectives, and its initialization overhead is negligible relative to fine-tuning time. EVA demonstrates robust performance gains and a clear pathway to more efficient fine-tuning of large foundation models.

Abstract

Foundation models (FMs) are pre-trained on large-scale datasets and then fine-tuned for a specific downstream task. The most common fine-tuning method is to update pretrained weights via low-rank adaptation (LoRA). Existing initialization strategies for LoRA often rely on singular value decompositions (SVD) of gradients or weight matrices. However, they do not provably maximize the expected gradient signal, which is critical for fast adaptation. To this end, we introduce Explained Variance Adaptation (EVA), an initialization scheme that uses the directions capturing the most activation variance, provably maximizing the expected gradient signal and accelerating fine-tuning. EVA performs incremental SVD on minibatches of activation vectors and selects the right-singular vectors for initialization once they converged. Further, by selecting the directions that capture the most activation-variance for a given rank budget, EVA accommodates adaptive ranks that reduce the number of trainable parameters. We apply EVA to a variety of fine-tuning tasks as language generation and understanding, image classification, and reinforcement learning. EVA exhibits faster convergence than competitors and achieves the highest average score across a multitude of tasks per domain while reducing the number of trainable parameters through rank redistribution. In summary, EVA establishes a new Pareto frontier compared to existing LoRA initialization schemes in both accuracy and efficiency.

Figures (16)

• Figure 1: Left: We perform incremental SVD on activation vectors for the first $T$ minibatches. Middle: We globally sort all right-singular vectors according to their explained variance given by their respective normalized singular values and only keep the top-k. Right: We allocate the top-k vectors as initialization for $\bm{A}$ and continue the standard LoRA fine-tuning procedure.
• Figure 2: Performance of all methods on eight common sense reasoning tasks (left) and MATH after being finetuned on MetaMathQA (right). EVA reduces the number of trainable parameters while reaching performance on-par or better.
• Figure 3: Performance of all methods for fine-tuning $\text{Llama-2-7B}$, $\text{Llama-3.1-8B}$, and $\text{Gemma-2-9B}$ on GSM8K after fine-tuning on the MetaMathQA dataset.
• Figure 4: Gradient norm (left) and training loss (right) for fine-tuning $\text{Llama-3.1-8B}$ on the MetaMathQA dataset. We compare EVA to other initialization methods and random initialization (LoRA). We show mean and standard deviation across three random seeds.
• Figure 5: Left: Percentage of training time required for computing data-driven initializations for $\text{Llama-2-7B}$ on a single A100 GPU on the common sense reasoning tasks. We report the maximum batch size and track peak memory usage. Right: Average cosine similarity between components after incremental SVD for different batch sizes. The components strongly correlate indicating that the SVD computation is mostly invariant to the batch size.

Theorems & Definitions (2)

• Theorem 3.1
• Theorem 3.2