MergeBench: A Benchmark for Merging Domain-Specialized LLMs

TL;DR

MergeBench provides a scalable, open benchmark for evaluating model merging on modern domain-specialized LLMs, spanning Llama-3 and Gemma backbones up to 9B parameters across five task domains. It standardizes base-model selection, finetuning, and evaluation, and compares eight merging algorithms across multi-task performance, forgetting, and runtime. Key findings show stronger base models yield higher merged performance, with coefficient tuning and sparsification helping preserve pretrained knowledge, though merging incurs non-trivial computational costs and in-domain performance can lag multi-task models in some settings. By releasing data, models, and evaluation pipelines, MergeBench establishes a foundation for reproducible, large-scale research into model merging and its practical deployment.

Abstract

Model merging provides a scalable alternative to multi-task training by combining specialized finetuned models through parameter arithmetic, enabling efficient deployment without the need for joint training or access to all task data. While recent methods have shown promise, existing evaluations are limited in both model scale and task diversity, leaving open questions about their applicability to large, domain-specialized LLMs. To tackle the challenges, we introduce MergeBench, a comprehensive evaluation suite designed to assess model merging at scale. MergeBench builds on state-of-the-art open-source language models, including Llama and Gemma families at 2B to 9B scales, and covers five key domains: instruction following, mathematics, multilingual understanding, coding and safety. We standardize finetuning and evaluation protocols, and assess eight representative merging methods across multi-task performance, forgetting and runtime efficiency. Based on extensive experiments, we provide practical guidelines for algorithm selection and share insights showing that model merging tends to perform better on stronger base models, with techniques such as merging coefficient tuning and sparsification improving knowledge retention. However, several challenges remain, including the computational cost on large models, the gap for in-domain performance compared to multi-task models, and the underexplored role of model merging in standard LLM training pipelines. We hope MergeBench provides a foundation for future research to advance the understanding and practical application of model merging. Our project page is at \href{https://yifei-he.github.io/mergebench/}{https://yifei-he.github.io/mergebench/}.

Figures (5)

• Figure 1: Overview of MergeBench. Starting from open-source base models (Llama and Gemma), we perform task-specific post-training on five diverse domains: mathematics, coding, multilinguality, instruction following, and safety. This process produces five task-specialized models that perform well on their respective domains but likely poorly on others. We then apply a range of model merging algorithms to combine these specialized models into a single multi-task model. MergeBench evaluates the effectiveness of these merging approaches along three key dimensions: multi-task performance, retention of pretrained knowledge (forgetting), and runtime efficiency.
• Figure 2: Normalized multi-task performance across base models. We report the average normalized performance of merged models relative to their corresponding specialized finetuned models. The four panels correspond to 2B&3B pretrained (top-left), 2B&3B instruction-tuned (top-right), 8B&9B pretrained (bottom-left), and 8B&9B instruction-tuned models (bottom-right), averaged over Gemma-2 and Llama-3 models of respective configurations. The dashed horizontal lines indicate the performance of base models prior to merging.
• Figure 3: Trade-off between generalization and multi-task performance (upper right better). Generalization is normalized by the base model performance, reflecting knowledge retention of the merged model. Left: averaged Llama performance. Right: averaged Gemma performance. Methods applying small merging coefficients or sparsification tend to incur less forgetting while maintaining competitive multi-task performance.
• Figure 4: Wall-clock time of different algorithms. The total time is broken into algorithm execution time (blue) and validation time for hyperparameter tuning (orange). This highlights the importance of considering validation overhead when assessing the practical efficiency of merging methods.
• Figure 5: Performance versus wall-clock time (upper left better). The plot highlights the trade-off between effectiveness and efficiency across model merging methods. Both versions of Localize-and-Stitch, RegMean, and Task Arithmetic achieve a favorable balance relative to other methods.