Spectral Surgery: Training-Free Refinement of LoRA via Gradient-Guided Singular Value Reweighting

TL;DR

Spectral Surgery is proposed, a training-free refinement that decomposes a LoRA update with SVD, estimates per-component sensitivity using gradients on a small calibration set, and reweights singular values under a magnitude constraint while keeping the learned directions fixed, demonstrating that SVD-structured, low-cost parameter editing can serve as a practical route to improving trained LoRA adapters in a purely post-hoc manner.

Abstract

Low-Rank Adaptation (LoRA) improves downstream performance by restricting task updates to a low-rank parameter subspace, yet how this limited capacity is allocated within a trained adapter remains unclear. Through a geometric and empirical study across multiple tasks and backbones, we find that trained LoRA updates often exhibit an inefficient spectrum: task effects concentrate in a small subset of singular directions, while many remaining components are neutral or detrimental, motivating post-hoc refinement within the learned subspace. We propose Spectral Surgery, a training-free refinement that decomposes a LoRA update with SVD, estimates per-component sensitivity using gradients on a small calibration set, and reweights singular values under a magnitude constraint while keeping the learned directions fixed. Across Llama-3.1-8B and Qwen3-8B on four benchmarks, Spectral Surgery yields consistent gains (up to +4.4 points on CommonsenseQA and +2.4 pass@1 on HumanEval) by adjusting only $\approx 1{,}000$ scalar coefficients. These results demonstrate that SVD-structured, low-cost parameter editing can serve as a practical route to improving trained LoRA adapters in a purely post-hoc manner.

Figures (8)

• Figure 1: Geometric structure of LoRA updates in residual-writing modules. We analyze LoRA updates of Qwen3-8B finetuned on the Alpaca dataset and visualize the alignment of the o_proj module in the shared residual output space. (a) The leading output direction ($u_1$) exhibits consistently high similarity across layers. (b) The full top-4 output subspace $U^{(4)}$ (with rank $r=16$) is also highly stable across layers, indicating a shared update manifold in the residual stream. (c) Within each layer, o_proj and down_proj are strongly aligned relative to a random-subspace baseline ($m/d_{\text{model}}$).
• Figure 2: Overview of Spectral Editing. We decompose the LoRA update $\Delta W$ into singular components ($U, \Sigma, V^\top$). We estimate a sensitivity score $s_k$ for each singular component using gradient projections on a calibration set, and then reweight the singular values in $\Sigma$ (via hard selection or continuous reweighting) to amplify task-relevant directions while suppressing noise. This reconstructs an edited update $\Delta W'$ without altering the singular subspaces.
• Figure 3: Guided vs. random perturbations. Each point is a (model, task) pair under the default setting. The x-axis is the improvement of random_index over the baseline, and the y-axis is the improvement of grad_direction. Points above the diagonal indicate genuine signal beyond random perturbation; the extreme failure on Qwen-IFEval illustrates the alignment tax of gradient-based editing.
• Figure 4: Safety trade-off of editing policies under the default setting. Reward is the mean improvement over aligned tasks (GSM8K, HumanEval, CSQA); Risk is the performance drop on the constraint-sensitive benchmark IFEval (clipped at 0 if a policy does not decrease IFEval). Gradient-based editing reaches the high-reward regime but can incur large risk, especially on IFEval.
• Figure 5: Llama-3.1-8B: Principal-direction similarity heatmap wall ($|u_1^\top u_1|$). Each cell shows the inter-layer similarity heatmap for a specific (task, module).