SimMerge: Learning to Select Merge Operators from Similarity Signals

TL;DR

This work tackles scalable model merging by reframing it as a predictive selection problem. SimMerge uses inexpensive pre-merge similarity signals to choose merge operators and, for multiway merges, merge orders, avoiding costly merge-and-evaluate searches. It learns from offline pairwise merges and generalizes to multiway plans and to a substantially larger 111B model without retraining, achieving a strong expert–auxiliary trade-off and demonstrating an effective online bandit variant for evolving catalogs. The results show that operator performance is regime-dependent and that per-instance predictions yield robust, scalable model composition under tight evaluation budgets.

Abstract

Model merging enables multiple large language models (LLMs) to be combined into a single model while preserving performance. This makes it a valuable tool in LLM development, offering a competitive alternative to multi-task training. However, merging can be difficult at scale, as successful merging requires choosing the right merge operator, selecting the right models, and merging them in the right order. This often leads researchers to run expensive merge-and-evaluate searches to select the best merge. In this work, we provide an alternative by introducing \simmerge{}, \emph{a predictive merge-selection method} that selects the best merge using inexpensive, task-agnostic similarity signals between models. From a small set of unlabeled probes, we compute functional and structural features and use them to predict the performance of a given 2-way merge. Using these predictions, \simmerge{} selects the best merge operator, the subset of models to merge, and the merge order, eliminating the expensive merge-and-evaluate loop. We demonstrate that we surpass standard merge-operator performance on 2-way merges of 7B-parameter LLMs, and that \simmerge{} generalizes to multi-way merges and 111B-parameter LLM merges without retraining. Additionally, we present a bandit variant that supports adding new tasks, models, and operators on the fly. Our results suggest that learning how to merge is a practical route to scalable model composition when checkpoint catalogs are large and evaluation budgets are tight.

Figures (17)

• Figure 1: Overview of SimMerge. Given a set of domain-specialized checkpoints and small unlabeled probe set for each domain, we compute pre-merge similarity signals, predict the merge operator for each binary merge step and the merge order, and then execute the selected plan once to obtain a single merged model.
• Figure 2: Fraction of the expert–auxiliary performance gap closed by each merge method across Code, Math, Multilingual, and RAG tasks. SimMerge consistently recovers a larger fraction of expert performance than fixed merge operators across all domains.
• Figure 3: Per-task percentage change in performance for each merge method. Blue markers show $\Delta_{\text{aux}}$ (change vs. auxiliary; higher is better) and red markers show $\Delta_{\text{expert}}$ (change vs. task expert; closer to $0$ indicates less degradation).
• Figure 4: Overall relative performance of 2-, 3-, and 4-way merges, reported as percentage change vs. the task expert (top) and vs. auxiliary models (bottom), macro-averaged over all tasks. SimMerge consistently improves over auxiliaries while limiting degradation relative to experts as the number of merged models increases.
• Figure 5: Performance of three-way merges across Code, Math, Multilingual, and RAG tasks, measured as the percentage of the expert–auxiliary performance gap that is closed. Auxiliary performance corresponds to 0%, expert performance to 100%.