LoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Merging
Zehua Liu, Han Wu, Yuxuan Yao, Ruifeng She, Xiongwei Han, Tao Zhong, Mingxuan Yuan
TL;DR
LoRE-Merging tackles the challenge of merging multiple fine-tuned LLMs without access to the base model by inducing an implicit, low-rank representation of task vectors. The method introduces an approximate base $\boldsymbol{\theta}_0$ and low-rank deltas $\boldsymbol{\delta}_i$, optimizing $f = \sum_{i=1}^n ( \| \boldsymbol{\theta}_0 + \boldsymbol{\delta}_i - \boldsymbol{\theta}_i \|_F^2 + \mu \| \boldsymbol{\delta}_i \|_*^2 )$ and solving with coordinate descent using Singular Value Thresholding (SVT). This yields a robust merging framework that can be followed by standard post-processing (e.g., Average Merging) to combine task vectors. Empirically, LoRE-Merging outperforms baselines on benchmarks like GSM8K, MATH, MMLU, GLUE, and MBPP across multiple LLM families, while demonstrating stability to hyperparameter variations and providing a practical approach for checkpoint-based and homogeneous merging scenarios. The work highlights a principled alternative to sparsity-based methods, with potential extensions to heterogeneous merging and broader task-composition settings.
Abstract
While most current approaches rely on further training techniques, such as fine-tuning or reinforcement learning, to enhance model capacities, model merging stands out for its ability of improving models without requiring any additional training. In this paper, we propose a unified framework for model merging based on low-rank estimation of task vectors without the need for access to the base model, named \textsc{LoRE-Merging}. Our approach is motivated by the observation that task vectors from fine-tuned models frequently exhibit a limited number of dominant singular values, making low-rank estimations less prone to interference. We implement the method by formulating the merging problem as an optimization problem. Extensive empirical experiments demonstrate the effectiveness of our framework in mitigating interference and preserving task-specific information, thereby advancing the state-of-the-art performance in model merging techniques.