Will it Merge? On The Causes of Model Mergeability

TL;DR

This work defines mergeability as the robustness of a model update’s knowledge when merged with other updates and formalizes it with a score $S(\theta_{\Delta})$, estimated through repeated merging trials. Across two experimental regimes (PopQA at the example level and Lots-of-LoRAs at the task level) using LoRA adapters and the Knots merging algorithm, the authors show that base-model task knowledge strongly predicts mergeability, while general domain knowledge and weight-level metrics are weak predictors. They also demonstrate that mergeability is largely an intrinsic property of the update rather than the merge set, and propose a weighted mean merging scheme using inverse base-model accuracy to better preserve weakly learned tasks without harming strong ones. The results suggest practical merging strategies that account for base-model familiarity to improve retention of knowledge across tasks, with implications for building robust multitask models from specialized adapters.

Abstract

Model merging has emerged as a promising technique for combining multiple fine-tuned models into a single multitask model without retraining. However, the factors that determine whether merging will succeed or fail remain poorly understood. In this work, we investigate why specific models are merged better than others. To do so, we propose a concrete, measurable definition of mergeability. We investigate several potential causes for high or low mergeability, highlighting the base model knowledge as a dominant factor: Models fine-tuned on instances that the base model knows better are more mergeable than models fine-tuned on instances that the base model struggles with. Based on our mergeability definition, we explore a simple weighted merging technique that better preserves weak knowledge in the base model.

Figures (42)

• Figure 1: Our Experimental setup. The figure shows an example from PopQA, Lots-of-LoRAs experiments follow a similar process. (a) We train an expert model for each example, and verify the model's correctness on the example after training. (b) We calculate each example's mergeability score, as defined in §\ref{['sec:mergeability']} and group examples by their mergeability score. (c) We investigate different traits that affect mergeability. Results are in §\ref{['sec:mergeability_causes']}. (d) We examine the correlation between the mergeability score and the evaluated properties. Among these, base model knowledge ($\Delta_{\text{base}}$) exhibits the strongest correlation.
• Figure 2: Mergeability score distribution of Llama-3.2-3B on the PopQA dataset. Blue wide bars show the mergeability score as empirically calculated. Red thin bars show the baseline distribution if mergability were not a model trait, modeled as a binomial distribution with a fixed success rate.
• Figure 3: PopQA different properties correlation with mergeability score results. Values are mean values normalized relative to $S=0.0$. $\Delta_{\text{base}}$ trend shows that the gap decreases with mergeability, implying that examples with better base model knowledge are more mergeable. Additionally, $\Delta_{\text{trained}}$ shows that high-mergeability examples achieve larger post-training probability improvements, although they were easier to "fix". Conversely, we observe no clear trend between mergeability score and training data difficulty (average perplexity and average context length) or weight-level properties (average weight norm and average highest singular value). The shaded regions represent the standard error of the values.
• Figure 4: Lots-of-LoRAs average base model task accuracy of different mergeability scores. Higher mergeability scores have on average a higher base model accuracy.
• Figure 5: The mergeability score when merging weights $\theta_\Delta$ with mergeability score $S(\theta_\Delta)=1.0$ with weights drawn from different mergeability scores. The blue line, which represents examples with a fixed score $S(\theta_\Delta)=1.0$, shows near-constant accuracy across conditions, while the orange line (other mergeability group examples) improves with their own mergeability score. The shaded regions represent the standard error of the accuracy.