Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

REAP the Experts: Why Pruning Prevails for One-Shot MoE compression

Mike Lasby, Ivan Lazarevich, Nish Sinnadurai, Sean Lie, Yani Ioannou, Vithursan Thangarasa

TL;DR

This work analyzes memory-efficient sparsely activated Mixture-of-Experts (SMoE) models and shows that pruning—when guided by a router-aware saliency criterion—outperforms expert merging for generative tasks. It proves that merging creates an irreducible error by collapsing the router's input-dependent control into a static target, with the bound scaling as $ ext{E}[(g_i+g_j)^2] \, ext{Var}[r(x)] \, \|\Delta_{ij}\|^2$, where $\Delta_{ij}=f_i-f_j$ and $r(x)=\tfrac{g_i(x)}{g_i(x)+g_j(x)}$, while pruning yields an error proportional to $ \mathbb{E}[g_j(x)^2] \, \|\Delta_{ij}\|^2$ and preserves independent control. The authors introduce REAP, with saliency $S_j = \frac{1}{|\mathcal{X}_j|} \sum_{x\in\mathcal{X}_j} g_j(x) \|f_j(x)\|_2$, to selectively prune low-impact experts across 6 architectures from 20B to 1T parameters. Empirically, REAP consistently outperforms merging and other pruning methods on generative benchmarks (code generation, writing, reasoning), achieving near-lossless coding performance at 50% compression on Qwen3-Coder-480B and Kimi-K2, and demonstrating robustness and domain-specific calibration advantages for deployment in resource-constrained settings.

Abstract

Sparsely-activated Mixture-of-Experts (SMoE) models offer efficient pre-training and low latency but their large parameter counts create significant memory overhead, motivating research into expert compression. Contrary to recent findings favouring expert merging on discriminative benchmarks, we demonstrate that expert pruning is a superior strategy for generative tasks. We prove that merging introduces an irreducible error by causing a "functional subspace collapse", due to the loss of the router's independent, input-dependent control over experts. Leveraging this insight, we propose Router-weighted Expert Activation Pruning (REAP), a novel pruning criterion that considers both router gate-values and expert activation norms. Across a diverse set of SMoE models ranging from 20B to 1T parameters, REAP consistently outperforms merging and other pruning methods on generative benchmarks, especially at 50% compression. Notably, our method achieves near-lossless compression on code generation and tool-calling tasks with Qwen3-Coder-480B and Kimi-K2, even after pruning 50% of experts.

REAP the Experts: Why Pruning Prevails for One-Shot MoE compression

TL;DR

This work analyzes memory-efficient sparsely activated Mixture-of-Experts (SMoE) models and shows that pruning—when guided by a router-aware saliency criterion—outperforms expert merging for generative tasks. It proves that merging creates an irreducible error by collapsing the router's input-dependent control into a static target, with the bound scaling as , where and , while pruning yields an error proportional to and preserves independent control. The authors introduce REAP, with saliency , to selectively prune low-impact experts across 6 architectures from 20B to 1T parameters. Empirically, REAP consistently outperforms merging and other pruning methods on generative benchmarks (code generation, writing, reasoning), achieving near-lossless coding performance at 50% compression on Qwen3-Coder-480B and Kimi-K2, and demonstrating robustness and domain-specific calibration advantages for deployment in resource-constrained settings.

Abstract

Sparsely-activated Mixture-of-Experts (SMoE) models offer efficient pre-training and low latency but their large parameter counts create significant memory overhead, motivating research into expert compression. Contrary to recent findings favouring expert merging on discriminative benchmarks, we demonstrate that expert pruning is a superior strategy for generative tasks. We prove that merging introduces an irreducible error by causing a "functional subspace collapse", due to the loss of the router's independent, input-dependent control over experts. Leveraging this insight, we propose Router-weighted Expert Activation Pruning (REAP), a novel pruning criterion that considers both router gate-values and expert activation norms. Across a diverse set of SMoE models ranging from 20B to 1T parameters, REAP consistently outperforms merging and other pruning methods on generative benchmarks, especially at 50% compression. Notably, our method achieves near-lossless compression on code generation and tool-calling tasks with Qwen3-Coder-480B and Kimi-K2, even after pruning 50% of experts.
Paper Structure (52 sections, 2 theorems, 21 equations, 8 figures, 7 tables)

This paper contains 52 sections, 2 theorems, 21 equations, 8 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Sparsely activated architecture.
  4. Expert pruning.
  5. Expert merging.
  6. Other compression techniques.
  7. Model merging.
  8. evaluation.
  9. Merging Experts Causes Functional Subspace Collapse
  10. Setup.
  11. Two operations at fixed compression.
  12. Merging induces an input-dependent target a single expert cannot realize
  13. Consequences.
  14. Pruning preserves independent control
  15. Synthesis.
  16. ...and 37 more sections

Key Result

Theorem 1

Let $\tilde{f}_\alpha(x)=\alpha f_i(x)+(1-\alpha)f_j(x)$ with a constant $\alpha\in[0,1]$ and define $\Delta_{ij}:=f_i(x)-f_j(x)$. The $L^2$ error of the merged pair is minimized when $\alpha$ is chosen to be the expected mixing ratio, $\alpha^\star:=\mathbb{E}[r(x)]$. Omitting the argument $(x)$ fo In particular, if the router's policy is not constant ($\mathrm{Var}[r(x)]>0$) and the experts are

Figures (8)

  • Figure 1: (\ref{['fig:qwen3-early']}) Functional subspace (PCA) for early layers in Qwen3-30B. Pruning (blue) preserves the manifold geometry; merging (green) collapses it toward the centre. (\ref{['fig:qwen3-late']}) Functional subspace (PCA) for late MoE layers. The contraction under merging is dramatically more pronounced, with up to 100$\times$ reduction in spread for models with many experts. See \ref{['fig:subspace-appendix']} for results from other models.
  • Figure 2: GLM-4.5-Air and Qwen3-30B accuracy vs. task type. offers significant improvements compared to other methods at 50% compression. Note the significant performance drop for merging methods on generative tasks (Coding, Math, Creative Writing) compared to their relative strength on MC.
  • Figure 3: (\ref{['fig:ngram']}) & (\ref{['fig:cross-ppl']}) N-Gram diversity and cross-perplexity of compressed Qwen3-30B-A3B models at 50% compression, respectively. (\ref{['fig:jsd']}) of compressed and baseline model logits vs. completion token position for Qwen3-30B-A3B at 50% compression. Initially, all compressed models share close alignment with the baseline model. However, as the completion token position increases the merged models diverge from the baseline more rapidly than the pruned model. (\ref{['fig:expert-dist']}) The mean relative L2-distance and singular-vector alignment between Qwen3-30B expert weights at 50% compression. Expert merging is more challenging than model merging due to large distances between experts in weight space and low singular-vector alignment.
  • Figure A4: (\ref{['fig:ernie-early']},\ref{['fig:mixtral-early']},\ref{['fig:llama-early']}) Functional subspace (PCA) for early layers. Pruning (blue) preserves the manifold geometry; merging (green) collapses it toward the centre. (\ref{['fig:ernie-late']},\ref{['fig:mixtral-late']},\ref{['fig:llama-late']}) Functional subspace (PCA) for late MoE layers.
  • Figure A5: Coding and accuracy across all models vs. parameters. The benefits of over other compression methods are evident at 50% compression. For large-scale , is near-lossless whereas the shortcomings of frequency-based pruning become apparent.
  • ...and 3 more figures

Theorems & Definitions (2)

  • Theorem 1: Irreducible error of merging
  • Theorem 2: Hierarchical clustering error