Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MergeMoE: Efficient Compression of MoE Models via Expert Output Merging

Ruijie Miao, Yilun Yao, Zihan Wang, Zhiming Wang, Bairen Yi, LingJun Liu, Yikai Zhao, Tong Yang

TL;DR

This work tackles the memory overhead of Mixture-of-Experts models by reframing expert merging as an output-merging problem and formulating it as an optimization task. It introduces MergeMoE, which constructs compression matrices through clustering, weighted averaging, and a least-squares solution to minimize forward-output error relative to the original MoE computation. The method yields theoretical grounding for the merging process and demonstrates consistent gains over baselines on multiple MoE models and NLP benchmarks at the same compression ratio, with minimal degradation compared to the full model. The approach offers practical, GPU-friendly compression with strong generalization across datasets and tasks, advancing efficient deployment of large MoE systems.

Abstract

The Mixture-of-Experts (MoE) technique has proven to be a promising solution to efficiently scale the model size, which has been widely applied in recent LLM advancements. However, the substantial memory overhead of MoE models has made their compression an important research direction. In this work, we provide a theoretical analysis of expert merging, a recently proposed technique for compressing MoE models. Rather than interpreting expert merging from the conventional perspective of parameter aggregation, we approach it from the perspective of merging experts' outputs. Our key insight is that the merging process can be interpreted as inserting additional matrices into the forward computation, which naturally leads to an optimization formulation. Building on this analysis, we introduce MergeMoE, a method that leverages mathematical optimization to construct the compression matrices. We evaluate MergeMoE on multiple MoE models and show that our algorithm consistently outperforms the baselines with the same compression ratios.

MergeMoE: Efficient Compression of MoE Models via Expert Output Merging

TL;DR

This work tackles the memory overhead of Mixture-of-Experts models by reframing expert merging as an output-merging problem and formulating it as an optimization task. It introduces MergeMoE, which constructs compression matrices through clustering, weighted averaging, and a least-squares solution to minimize forward-output error relative to the original MoE computation. The method yields theoretical grounding for the merging process and demonstrates consistent gains over baselines on multiple MoE models and NLP benchmarks at the same compression ratio, with minimal degradation compared to the full model. The approach offers practical, GPU-friendly compression with strong generalization across datasets and tasks, advancing efficient deployment of large MoE systems.

Abstract

The Mixture-of-Experts (MoE) technique has proven to be a promising solution to efficiently scale the model size, which has been widely applied in recent LLM advancements. However, the substantial memory overhead of MoE models has made their compression an important research direction. In this work, we provide a theoretical analysis of expert merging, a recently proposed technique for compressing MoE models. Rather than interpreting expert merging from the conventional perspective of parameter aggregation, we approach it from the perspective of merging experts' outputs. Our key insight is that the merging process can be interpreted as inserting additional matrices into the forward computation, which naturally leads to an optimization formulation. Building on this analysis, we introduce MergeMoE, a method that leverages mathematical optimization to construct the compression matrices. We evaluate MergeMoE on multiple MoE models and show that our algorithm consistently outperforms the baselines with the same compression ratios.
Paper Structure (35 sections, 2 theorems, 32 equations, 5 figures, 6 tables)

This paper contains 35 sections, 2 theorems, 32 equations, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Mixture-of-Experts models.
  4. Model Compression.
  5. Background and Theoretical Insights
  6. Preliminary
  7. Insights for Expert Merging
  8. M-SMoE under Our Output-Merging View
  9. Methodology
  10. Summary of the algorithm design.
  11. Evaluation
  12. Setup
  13. Models and Datasets.
  14. Evaluation Details.
  15. Performance of MergeMoE
  16. ...and 20 more sections

Key Result

Theorem 1

Given $A\in \mathbb{R}^{M\times N}$, $Y_0\in \mathbb{R}^{K\times N}$, each column of $A$ has exactly one $1$ and the rest are $0$. Let $B\in \mathbb{R}^{N\times M}$, $v_1,v_2,...,v_M$ be the columns of $B$. Let $C_i$ be the indices corresponding to the non-zero values of the $i$-th column of $A$. Fo is a minimal point of the function:

Figures (5)

  • Figure 1: An overview of how the merging algorithm changes the forward process of the MoE module. It shows the transition from an initial $8$-expert configuration (top-$2$ activation per token) to $4$ experts after compresion.
  • Figure 2: Experiments on the effects of different compression ratios.
  • Figure 3: Comparison of the time cost.
  • Figure 4: Evaluation on the impact of the number of sample size.
  • Figure 5: Evaluation on the IFEval benchmark.

Theorems & Definitions (3)

  • Theorem 1
  • Theorem 1
  • proof