Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations

TL;DR

This work tackles the memory overhead of ultra-large Mixture-of-Experts models by showing that domain-specific experts can be identified with few demonstrations and pruned with minimal performance loss. It introduces EASY-EP, a pruning framework that combines output-aware expert importance and token-level contribution to select a small, highly relevant subset of experts in a single forward pass. Empirical results on DeepSeek-R1 and DeepSeek-V3-0324 demonstrate that pruning about half the experts preserves near full-domain performance, while achieving up to $2.99\times$ throughput under the same memory budget, and that mixed-domain pruning maintains over 90% of the original performance. The approach enables memory-efficient deployment of large MoE models by leveraging domain specialization without retraining, with broader implications for scalable inference in domain-rich AI systems.

Abstract

Mixture-of-Experts (MoE) models achieve a favorable trade-off between performance and inference efficiency by activating only a subset of experts. However, the memory overhead of storing all experts remains a major limitation, especially in large-scale MoE models such as DeepSeek-R1(671B). In this study, we investigate domain specialization and expert redundancy in large-scale MoE models and uncover a consistent behavior we term few-shot expert localization, with only a few in-domain demonstrations, the model consistently activates a sparse and stable subset of experts on tasks within the same domain. Building on this observation, we propose a simple yet effective pruning framework, EASY-EP, that leverages a few domain-specific demonstrations to identify and retain only the most relevant experts. EASY-EP comprises two key components: output-aware expert importance assessment and expert-level token contribution estimation. The former evaluates the importance of each expert for the current token by considering the gating scores and L2 norm of the outputs of activated experts, while the latter assesses the contribution of tokens based on representation similarities before and after routed experts. Experiments on DeepSeek-R1 and DeepSeek-V3-0324 show that our method can achieve comparable performances and $2.99\times$ throughput under the same memory budget with full model with only half the experts.

Figures (11)

• Figure 1: Throughput and performance comparison of DeepSeek-R1 on AIME2024 with varying expert numbers using EASY-EP. We deploy DeepSeek-R1 with two 8$\times$ H800 for 224 and 256 experts, while one $8\times$ H800 for others. The throughputs of the latter configurations are multiplied by 2.
• Figure 1: Results of removing domain-specific experts. Bold denotes in-domain results.
• Figure 2: (A), (B): Overlap ratios of experts with top gating scores on different datasets. (C): Overlap ratio of top-128 experts with different numbers of demonstrations. (D): Overlap ratio of top-128 experts pruned with different math datasets.
• Figure 3: Overall framework of EASY-EP. Given a calibration set consisting of input and responses by the model, EASY-EP leverages output-aware expert importance assessment and expert-level token contribution estimation to compute the expert score on the domain and returns the pruned expert sets.
• Figure 4: Comparison of performance with different numbers of shots for pruning.