Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniques

TL;DR

This work tackles the inefficiencies of Mixture of Experts (MoE) models by proposing a holistic compression framework that combines Expert Trimming (Layer Drop, Block Drop) with Expert Slimming (pruning/quantization). The approach reduces both model size and runtime overhead while preserving strong performance, achieving up to 6.05x speedups and substantial memory reductions on Mixtral-8×7B, with post-finetuning further closing any remaining gaps. Key insights include the greater redundancy of MoE layers compared to dense FFNs and the complementary benefits of coarse-grained pruning plus per-expert slimming, all backed by extensive experiments and visualizations. The work provides a practical recipe and a code release to advance scalable deployment of MoE-based systems.

Abstract

Scaling large language models has driven remarkable advancements across various domains, yet the continual increase in model size presents significant challenges for real-world deployment. The Mixture of Experts (MoE) architecture offers a promising solution by dynamically selecting and activating only a subset of experts during inference, thus substantially reducing computational costs while preserving high performance. Despite these benefits, MoE introduces new inefficiencies, such as excessive parameters and communication overhead. In this work, we present a holistic study of compression techniques for Mixture of Experts to enhance both efficiency and scalability. While recent efforts have focused on Expert Trimming, which reduces the number of experts, these approaches still suffer from considerable communication and computational costs. To address this, we propose more aggressive strategies, such as Layer Drop, which removes entire MoE layers, and Block Drop, which eliminates transformer blocks. Surprisingly, these aggressive pruning techniques not only preserve model performance but also substantially improve computation and memory efficiency. Furthermore, beyond Expert Trimming, we also introduce Expert Slimming, which compresses individual experts to further boost performance and can be seamlessly integrated with Expert Trimming. Extensive experimental results demonstrate the effectiveness of our proposed methods-Layer Drop and Block Drop-along with the comprehensive recipe that integrates Expert Slimming and Expert Trimming, achieving a 6.05x speedup with 77.1% reduced memory usage while maintaining over 92% of performance on Mixtral-8x7B. Our code is released at https://github.com/CASE-Lab-UMD/Unified-MoE-Compression.

Figures (15)

• Figure 1: The Unified View of MoE Compression. The view integrates two complementary perspectives: Expert Slimming and Expert Trimming. Expert Slimming compresses individual experts, while Expert Trimming directly drops structured modules.
• Figure 2: Illustration of Similarity Measurements in Layer Drop. Features for calculating $\boldsymbol{S}^{\text{(M)}}$ and $\boldsymbol{S}^{\text{(NM)}}$ are colored with red and blue, respectively.
• Figure 3: Evaluation of Expert Drop. We consider two strategies: layer-wise dropping (dotted lines) and global dropping (solid lines). "Random Guess" refers to randomly generating an output rather than using the model’s predictions, serving as a baseline to assess the extent of performance degradation.
• Figure 4: Layer-Wise Similarity. We consider two scenarios, i.e., for "MoE" and "Norm + MoE".
• Figure 5: Evaluation of Layer Drop. We show results on Mixtral-8$\times$7B and DeepSeek-MoE-16B (solid lines), along with the baseline and random guess performances (dotted lines).