Towards Reversible Model Merging For Low-rank Weights

TL;DR

This paper tackles the problem that conventional merging fails to preserve performance when merging low-rank adapters from LoRA or PT-SVD. It introduces Reversible Model Merging (RMM), which stores a compact basis and per-task coefficients to reconstruct each task’s low-rank deltas, rather than collapsing all tasks into a single model. The method solves a closed-form basis selection problem, obtaining the top-$p$ right singular vectors as the basis and using least-squares to obtain task coefficients, with a reconstruction that includes a per-position mean term. Empirically, RMM consistently outperforms existing merging approaches across multiple tasks and compression regimes, achieving substantial accuracy retention while offering a tunable storage-performance trade-off and favorable scalability for many tasks.

Abstract

Model merging aims to combine multiple fine-tuned models into a single set of weights that performs well across all source tasks. While prior work has shown that merging can approximate the performance of individual fine-tuned models for each task, it largely overlooks scenarios where models are compressed into low-rank representations, either through low-rank adaptation (LoRA) or post-training singular value decomposition (SVD). We first demonstrate that applying conventional merging methods to low-rank weights leads to severe performance degradation in the merged model. Motivated by this phenomenon, we propose a fundamentally different approach: instead of collapsing all adapters into one set of weights, we construct a compact basis (e.g., an equivalent of holding two or more models) from which original task-specific models can be recovered via linear combination. This reframes merging as generating a reconstruction-capable model space rather than producing a single merged model. Crucially, this allows us to ``revert'' to each individual model when needed, recognizing that no merged model can consistently outperform one specialized for its task. Building on this insight, we introduce our method, Reversible Model Merging (RMM), an efficient, data-free, and flexible method that provides a closed-form solution for selecting the optimal basis of model weights and task-specific coefficients for linear combination. Extensive experiments across diverse datasets and model scales demonstrate that RMM consistently outperforms existing merging approaches, preserving the performance of low-rank compressed models by a significant margin.

Figures (5)

• Figure 1: Average task performance of individual models (before merging) versus merged models for different merging methods. Results are shown for fully fine-tuned models and models fine-tuned using LoRA with various ranks $r\in \{16,32,64,128\}$.
• Figure 2: Average task performance of individual models (before merging) versus merged models for different merging methods. Results are shown for fully fine-tuned models and models compressed using PT-SVD with various ranks $r\in \{16,32,64,128\}$.
• Figure 3: Overview of the proposed RMM framework. Left: During the merging phase, we compute a compact set of basis directions (e.g., two in this example) and their associated task-specific coefficients. Right: At inference, a target model $i$ is reconstructed by linearly combining the basis vectors using the corresponding coefficients.
• Figure 4: Scalability of RMM ($p=2$). The storage ratio indicates the percentage of memory required relative to storing all models.
• Figure 5: Scalability of RMM ($p=3$). The storage ratio indicates the percentage of memory required relative to storing all models.

Theorems & Definitions (1)

• Theorem 1: Optimal Solution to equation \ref{['objec']}