A Survey on Inference Optimization Techniques for Mixture of Experts Models

TL;DR

MoE models offer scalable capacity via sparse, gated routing but pose unique inference challenges in latency, memory, and energy use. The paper delivers a structured survey across model-, system-, and hardware-level optimizations, detailing techniques from efficient architecture and compression to distributed execution and accelerator design. It synthesizes methods like dynamic gating, expert pruning/quantization/distillation, offloading, and co-design strategies, and highlights representative systems and benchmarks. By mapping the landscape and providing a public repository, the work guides researchers and practitioners toward effective MoE deployment across cloud and edge environments.

Abstract

The emergence of large-scale Mixture of Experts (MoE) models represents a significant advancement in artificial intelligence, offering enhanced model capacity and computational efficiency through conditional computation. However, deploying and running inference on these models presents significant challenges in computational resources, latency, and energy efficiency. This comprehensive survey analyzes optimization techniques for MoE models across the entire system stack. We first establish a taxonomical framework that categorizes optimization approaches into model-level, system-level, and hardware-level optimizations. At the model level, we examine architectural innovations including efficient expert design, attention mechanisms, various compression techniques such as pruning, quantization, and knowledge distillation, as well as algorithm improvement including dynamic routing strategies and expert merging methods. At the system level, we investigate distributed computing approaches, load balancing mechanisms, and efficient scheduling algorithms that enable scalable deployment. Furthermore, we delve into hardware-specific optimizations and co-design strategies that maximize throughput and energy efficiency. This survey provides both a structured overview of existing solutions and identifies key challenges and promising research directions in MoE inference optimization. To facilitate ongoing updates and the sharing of cutting-edge advances in MoE inference optimization research, we have established a repository accessible at https://github.com/MoE-Inf/awesome-moe-inference/.

Figures (8)

• Figure 1: Taxonomy of MoE inference optimization.
• Figure 2: Architectural comparison of dense layer with MoE layers: (a) conventional dense transformer layer, (b) transformer layer with MoE-based feed-forward network, and (c) transformer layer with both MoE-based attention and feed-forward networks.
• Figure 3: Model-level inference optimization techniques for MoE.
• Figure 4: Compression techniques for MoE models.
• Figure 5: Algorithm improvement on expert layers.