Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Faster MoE LLM Inference for Extremely Large Models

Haoqi Yang, Luohe Shi, Qiwei Li, Zuchao Li, Ping Wang, Bo Du, Mengjia Shen, Hai Zhao

TL;DR

This paper analyzes inference efficiency for extremely large, fine-grained MoE LLMs, comparing them to traditional coarse MoE approaches and focusing on how service load and expert activation affect throughput and accuracy. It shows that reducing the number of activated experts can yield notable throughput gains with limited performance loss, while pruning the total number of experts offers diminishing returns and potential degradation. The authors evaluate strategies including inference-time expert skipping and pre-inference expert pruning, finding that softmax-based routing with careful layer-wise reductions maintains performance better than more aggressive or random pruning, and that expert parallelism can substantially reduce inter-device communication. Overall, the work demonstrates substantial optimization opportunities for MoE inference, including practical gains of at least 10% throughput in some setups and up to 2.3× speedups under low concurrency, highlighting Fine-grained MoE as a fertile area for scalable, efficient LLM deployment.

Abstract

Sparse Mixture of Experts (MoE) large language models (LLMs) are gradually becoming the mainstream approach for ultra-large-scale models. Existing optimization efforts for MoE models have focused primarily on coarse-grained MoE architectures. With the emergence of DeepSeek Models, fine-grained MoE models are gaining popularity, yet research on them remains limited. Therefore, we want to discuss the efficiency dynamic under different service loads. Additionally, fine-grained models allow deployers to reduce the number of routed experts, both activated counts and total counts, raising the question of how this reduction affects the trade-off between MoE efficiency and performance. Our findings indicate that while deploying MoE models presents greater challenges, it also offers significant optimization opportunities. Reducing the number of activated experts can lead to substantial efficiency improvements in certain scenarios, with only minor performance degradation. Reducing the total number of experts provides limited efficiency gains but results in severe performance degradation. Our method can increase throughput by at least 10\% without any performance degradation. Overall, we conclude that MoE inference optimization remains an area with substantial potential for exploration and improvement.

Faster MoE LLM Inference for Extremely Large Models

TL;DR

This paper analyzes inference efficiency for extremely large, fine-grained MoE LLMs, comparing them to traditional coarse MoE approaches and focusing on how service load and expert activation affect throughput and accuracy. It shows that reducing the number of activated experts can yield notable throughput gains with limited performance loss, while pruning the total number of experts offers diminishing returns and potential degradation. The authors evaluate strategies including inference-time expert skipping and pre-inference expert pruning, finding that softmax-based routing with careful layer-wise reductions maintains performance better than more aggressive or random pruning, and that expert parallelism can substantially reduce inter-device communication. Overall, the work demonstrates substantial optimization opportunities for MoE inference, including practical gains of at least 10% throughput in some setups and up to 2.3× speedups under low concurrency, highlighting Fine-grained MoE as a fertile area for scalable, efficient LLM deployment.

Abstract

Sparse Mixture of Experts (MoE) large language models (LLMs) are gradually becoming the mainstream approach for ultra-large-scale models. Existing optimization efforts for MoE models have focused primarily on coarse-grained MoE architectures. With the emergence of DeepSeek Models, fine-grained MoE models are gaining popularity, yet research on them remains limited. Therefore, we want to discuss the efficiency dynamic under different service loads. Additionally, fine-grained models allow deployers to reduce the number of routed experts, both activated counts and total counts, raising the question of how this reduction affects the trade-off between MoE efficiency and performance. Our findings indicate that while deploying MoE models presents greater challenges, it also offers significant optimization opportunities. Reducing the number of activated experts can lead to substantial efficiency improvements in certain scenarios, with only minor performance degradation. Reducing the total number of experts provides limited efficiency gains but results in severe performance degradation. Our method can increase throughput by at least 10\% without any performance degradation. Overall, we conclude that MoE inference optimization remains an area with substantial potential for exploration and improvement.
Paper Structure (32 sections, 3 equations, 6 figures, 12 tables)

This paper contains 32 sections, 3 equations, 6 figures, 12 tables.

Table of Contents

  1. Introduction
  2. Background and Related Works
  3. FFN and MoE
  4. LLM Serving Efficiency
  5. Model Pruning
  6. Preliminaries
  7. Notations
  8. Evaluation Method
  9. Hardware and Implementation Details
  10. Severing Efficiency of MoE
  11. Weakened Batch Effect
  12. Expert Parallel
  13. Inference Time Expert Skipping
  14. Efficiency
  15. Performance and Structure Searching
  16. ...and 17 more sections

Figures (6)

  • Figure 1: The comparison of FFN, coarse-grained MoE, and fine-grained MoE.
  • Figure 2: Simulation experiment results. X-axis represents sequence length $L$.
  • Figure 3: How expert skipping influence throughput. X-axis represents the concurrency.
  • Figure 4: How expert skipping influence performance
  • Figure 5: Throughput and speedup of expert pruning.
  • ...and 1 more figures