Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models

TL;DR

This work addresses deployment inefficiencies of Mixture-of-Experts LLMs by introducing hardware-friendly, post-training expert pruning and dynamic expert skipping. The proposed methods prune unused experts per layer and selectively skip underutilized experts during inference, reducing memory footprint and increasing speed without retraining. Empirical results on Mixtral 8x7B show substantial memory reductions and 1.2–1.3x speedups, with modest task-agnostic performance losses that can be mitigated via domain-specific calibration and fine-tuning. Domain-focused pruning using MATH data yields significant gains on mathematics tasks, and combining pruning with dynamic skipping delivers favorable speed-accuracy trade-offs, highlighting practical pathways for efficient MoE LLM deployment.

Abstract

A pivotal advancement in the progress of large language models (LLMs) is the emergence of the Mixture-of-Experts (MoE) LLMs. Compared to traditional LLMs, MoE LLMs can achieve higher performance with fewer parameters, but it is still hard to deploy them due to their immense parameter sizes. Different from previous weight pruning methods that rely on specifically designed hardware, this paper mainly aims to enhance the deployment efficiency of MoE LLMs by introducing plug-and-play expert-level sparsification techniques. Specifically, we propose, for the first time to our best knowledge, post-training approaches for task-agnostic and task-specific expert pruning and skipping of MoE LLMs, tailored to improve deployment efficiency while maintaining model performance across a wide range of tasks. Extensive experiments show that our proposed methods can simultaneously reduce model sizes and increase the inference speed, while maintaining satisfactory performance. Data and code will be available at https://github.com/Lucky-Lance/Expert_Sparsity.

Figures (5)

• Figure 1: Memory usage reduction (bf16) and inference speedup illustration of our proposed post-training expert pruning and dynamic (expert) skipping methods on the Mixtral 8x7B jiang2024mixtral model. Our method greatly reduces memory consumption and enhances inference speed.
• Figure 2: Illustration of the MoE layer in the Mixtral 8x7B model for per-token inference. The output of the layer is the weighted sum of the outputs from selected experts over input token $\boldsymbol{x}$. $\widetilde{w}_i$ denotes the normalized routing weight of each selected expert.
• Figure 3: Framework of our proposed expert pruning and dynamic skipping methods. (a) The expert pruning method evaluates the contributions of experts via a small calibration dataset and then permanently discards those with low contributions (e.g., experts with a slashed background). (b) The dynamic skipping method discard no experts instead dynamically decides whether to skip certain experts (e.g., experts with a yellow background) during inference.
• Figure 4: Expert selection comparison between C4 and MATH dataset with $r=6$ for Mixtral 8x7B model. Significant divergence is observed in the selection of experts across these two datasets, and identical expert combinations are observed in only four specific layers (i.e., layer 2, layer 4, layer16, and layer 31).
• Figure 5: Frequency visualization of expert selection in layer 0, layer 15, and layer 31 for the Mixtral 8x7B model on samples of (a) C4 2019t5 and (b) MATH hendrycksmath2021 dataset respectively. The model exhibits certain preferences in the selection of experts.