SRLoRA: Subspace Recomposition in Low-Rank Adaptation via Importance-Based Fusion and Reinitialization

TL;DR

SRLoRA addresses the expressiveness bottleneck of Low-Rank Adaptation by introducing a dynamic subspace recomposition mechanism. It performs importance-based fusion of underutilized LoRA components into the frozen backbone and reinitializes them along unused principal directions derived from the pretrained weight's SVD, preserving the parameter budget. The approach yields faster convergence and improved accuracy across language and vision benchmarks, including GLUE and ViT-based image classification, demonstrating broad applicability to PEFT. By continually refreshing the adaptation subspace without increasing trainable parameters, SRLoRA offers a practical, general enhancement to low-rank fine-tuning with potential for broader PEFT use.

Abstract

Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning (PEFT) method that injects two trainable low-rank matrices (A and B) into frozen pretrained models. While efficient, LoRA constrains updates to a fixed low-rank subspace (Delta W = BA), which can limit representational capacity and hinder downstream performance. We introduce Subspace Recomposition in Low-Rank Adaptation (SRLoRA) via importance-based fusion and reinitialization, a novel approach that enhances LoRA's expressiveness without compromising its lightweight structure. SRLoRA assigns importance scores to each LoRA pair (a column of B and the corresponding row of A), and dynamically recomposes the subspace during training. Less important pairs are fused into the frozen backbone, freeing capacity to reinitialize new pairs along unused principal directions derived from the pretrained weight's singular value decomposition. This mechanism enables continual subspace refreshment and richer adaptation over time, without increasing the number of trainable parameters. We evaluate SRLoRA on both language and vision tasks, including the GLUE benchmark and various image classification datasets. SRLoRA consistently achieves faster convergence and improved accuracy over standard LoRA, demonstrating its generality, efficiency, and potential for broader PEFT applications.

Figures (5)

• Figure 1: Comparison with LoRA and SRLoRA. The blue areas represent pretrained/frozen parameters during training. Left: LoRA. The orange areas represent trainable parameters; Right: SRLoRA. Darker orange areas represent higher importance scores, and lighter orange areas represent lower importance scores.
• Figure 2: Overview of SRLoRA. (a) SVD of pretrained weights. (b) Fusion of low-importance LoRA components into the base model. Lighter orange areas indicate lower importance scores, representing components that are being fused into the base model.
• Figure 3: Training loss comparison on RTE, QNLI, SST2 and CoLA tasks. SRLoRA demonstrates faster convergence and improved stability over baseline methods.
• Figure 4: Variance of activation intervals across different candidate ranks for each SRLoRA-enabled layer in DeBERTa-v3-base on the CoLA task. "Active intervals" refer to the rank positions that receive significant updates during low-rank adaptation. Lower variance in the Feedforward (FFN) layers suggests that many rank positions are equally important, indicating a need to update a broader subspace. In contrast, attention projection layers (Q, K, V) show high variance, meaning only a few rank directions dominate, aligned with the presence of large singular values.
• Figure 5: Active intervals for target ranks in Layer 3 of DeBERTa-v3-base on the CoLA task. Each interval represents the duration (in training steps or epochs) that a given SRLoRA rank remains actively updated before reverting to the frozen base weights. Ranks corresponding to larger singular values (shown on the left) tend to stay active longer, indicating their greater importance in adaptation. In contrast, dense (FFN) layers exhibit a more evenly distributed activation pattern, with ranks switching more frequently and balanced across the subspace.