Accurate and Efficient Low-Rank Model Merging in Core Space

TL;DR

This work tackles the challenge of merging multiple LoRA-based adaptations without sacrificing the efficiency of low-rank updates. It introduces Core Space merging, a lossless shared subspace built from reference bases derived via SVD, enabling compact, aligned representations $M^{(t)}$ for each task and allowing merges to be performed in a reduced $Tr\times Tr$ space. The framework preserves information, offers favorable computational complexity, and yields state-of-the-art results across vision and language benchmarks while delivering substantial speedups over existing methods. Its compatibility with various merging strategies and extension to VeRA highlight its practicality for scalable, multi-task fine-tuning of large models.

Abstract

In this paper, we address the challenges associated with merging low-rank adaptations of large neural networks. With the rise of parameter-efficient adaptation techniques, such as Low-Rank Adaptation (LoRA), model fine-tuning has become more accessible. While fine-tuning models with LoRA is highly efficient, existing merging methods often sacrifice this efficiency by merging fully-sized weight matrices. We propose the Core Space merging framework, which enables the merging of LoRA-adapted models within a common alignment basis, thereby preserving the efficiency of low-rank adaptation while substantially improving accuracy across tasks. We further provide a formal proof that projection into Core Space ensures no loss of information and provide a complexity analysis showing the efficiency gains. Extensive empirical results demonstrate that Core Space significantly improves existing merging techniques and achieves state-of-the-art results on both vision and language tasks while utilizing a fraction of the computational resources. Codebase is available at https://github.com/apanariello4/core-space-merging.

Figures (6)

• Figure 1: Merging in full space is fast but suboptimal (bottom center). Merging in KnOTS space or using strong merging methods (e.g., TSV) improves performance but increases cost by orders of magnitude (right). Core Space merging is effective and efficient (top left). Results on Llama 3 8B.
• Figure 2: Full Space Merging (left) firstly reconstructs full space matrices $\Delta W^{(t)} = B^{(t)} A^{(t)}$, and then performs merging in the full space to obtain $\Delta W$. KnOTS Merging concatenates the $\Delta W^{(t)}$ matrices, and performs a costly SVD on this high-dimensional matrix. Then, the $V^{(t)}$ matrices are merged and used to obtain the final $\Delta W$. The proposed Core Space Merging (right) performs SVD on a concatenation of low-dimensional $A^{(t)}$ and $B^{(t)}$ matrices to obtain reference bases $(V_{A}^{\mathrm{ref}}, U_{B}^{\mathrm{ref}})$. Afterwards, it projects each update into the Core Space to obtain the core matrices $\{M^{(t)}\}_{t=1}^T$. It then performs merging in the Core Space and reconstructs to obtain the final $\Delta W$.
• Figure 3: Core Space merging is more efficient than the previous state-of-the-art KnOTS. The cost is similar to full space merging, which results in much lower performance. We visualize the number of operations performed to merge $T$ rank $r$ LoRA modules of final shape $n \times n$.
• Figure 4: Most components in full space are irrelevant when doing Task-Arithmetic (TA). Removing any components from the core space results in a performance drop, showing that it is an information-dense space. We report the results on vision tasks with ViT-B/32.
• Figure 5: Subspace Alignment Ratio (SAR) marczak2025task. Each bar shows the average SAR between LoRA task matrices, Full, and Core Space. In Core Space, task matrices exhibit higher SAR. The associated performance gains suggest that better alignment facilitates more effective merging.

Theorems & Definitions (9)

• Definition 1: Reference Bases
• Definition 2: Core Matrix
• Lemma
• Lemma
• proof
• Lemma
• proof
• Lemma
• proof