MergeME: Model Merging Techniques for Homogeneous and Heterogeneous MoEs

TL;DR

This work tackles the challenge of merging domain-specific expert LLMs into a single Mixture-of-Experts (MoE) by addressing parameter interference and architecture divergence. It introduces three core advances: (i) advanced merging methods (Dare and Ties) to mitigate interference in homogeneous MoEs, (ii) routing heuristics based on perplexity to reduce dependency on fine-tuning, and (iii) a projector-based framework to merge heterogeneous experts via a sequence-level router. Across math, coding, and knowledge domains, the proposed methods outperform state-of-the-art BTX baselines and demonstrate effective heterogeneous MoE merging, with reduced fine-tuning costs and broader applicability. These results significantly broaden practical MoE merging, enabling efficient domain specialization and potential extension to larger models and multimodal setups.

Abstract

The recent success of specialized Large Language Models (LLMs) in domains such as mathematical reasoning and coding has led to growing interest in methods for merging these expert LLMs into a unified Mixture-of-Experts (MoE) model, with the goal of enhancing performance in each domain while retaining effectiveness on general tasks. However, the effective merging of expert models remains an open challenge, especially for models with highly divergent weight parameters or different architectures. State-of-the-art MoE merging methods only work with homogeneous model architectures and rely on simple unweighted averaging to merge expert layers, which does not address parameter interference and requires extensive fine-tuning of the merged MoE to restore performance. To address these limitations, this paper introduces new MoE merging techniques, including strategies to mitigate parameter interference, routing heuristics to reduce the need for MoE fine-tuning, and a novel method for merging experts with different architectures. Extensive experiments across multiple domains demonstrate the effectiveness of our proposed methods, reducing fine-tuning costs, improving performance over state-of-the-art methods, and expanding the applicability of MoE merging.

Figures (10)

• Figure 1: Overview of the proposed MoE framework for homogeneous model merging. We replace averaging with Dare or Ties merging to reduce parameter interference. Additionally, we introduce novel routing heuristics to enhance performance without fine-tuning.
• Figure 2: Different types of parameter interference and merged outputs produced by simple averaging.
• Figure 3: Overview of the proposed MoE framework for heterogeneous experts. Each color represents one heterogeneous expert. $n_1, \cdots, n_4$ refers to the number of layers in each expert.
• Figure 4: Similarity of task vector for attention and FFNs layers for Math and Code Expert experts. We average the similarity of attentions or FFNs in one decoder layers as the overall similarity for each layer.
• Figure 5: Routing probability of experts on GSM8K and MATH for different merging methods.