Toward Cost-Efficient Serving of Mixture-of-Experts with Asynchrony

TL;DR

MoE inference suffers from load skew that causes GPU idle time and barrier-induced stalls. The paper presents Asynchronous Expert Parallelism with $μ$-queuing and adaptive re-batching, implemented in AMoE to decouple layer execution from synchronization. It introduces a defragging scheduler, token metadata management, and a disaggregated architecture that yields up to 3x throughput and near-linear multi-node scalability. This approach enables cost-efficient, scalable serving for large-scale MoE models by reducing idle time and improving hardware utilization.

Abstract

Mixture-of-Experts (MoE) architectures offer the promise of larger model capacity without the prohibitive costs of fully dense designs. However, in real-world inference serving, load skew across experts often leads to suboptimal device utilization and excessive synchronization overheads. This paper introduces Asynchronous Expert Parallelism (AEP), a new paradigm that decouples layer execution from barrier-style synchronization. By dynamically queuing tokens at each layer (referred to as $μ$-queuing) and adaptively re-batching them on demand, GPUs avoid waiting for straggling experts and instead continuously process whichever layer is ready. This asynchronous approach mitigates two major inefficiencies in traditional expert-parallel systems: (1) idle GPU time while waiting for the hottest expert, and (2) small-batch executions on colder experts that waste memory bandwidth. We implement these ideas in a serving system called AMoE, which disaggregates attention from expert layers and uses a defragging scheduler to reduce batch fragmentation. Evaluations on prototype MoE models show that AMoE improves throughput by up to 2.7x compared to state-of-the-art baselines, incurring a manageable latency penalty and providing a cost-effective operating point. Furthermore, experiments demonstrate nearly linear scalability to multi-node settings, whereas the baseline system shows no throughput increase even when the number of GPUs is doubled.

Figures (18)

• Figure 1: Expert load skews causes inefficient GPU executions.
• Figure 2: Asynchronous Expert Parallelism. Schedulers in each GPU freely selects layer to execute with accumulated tokens.
• Figure 3: Execution throughput of a single expert layer with different batch sizes on A100 40GB.
• Figure 4: (a) expert load skew of a single iteration and (b) resulting GPU stall time fraction while serving Mixtral 8x7B with databricks-dolly-15k dataset at 100 req/s input rate on DGX A100 40GB (8x A100 40G with NVSwitch) using SGLang with expert parallelism. Mixtral 8x7B has 32 decoding blocks and 8 experts per block.
• Figure 5: System architecture of AMoE.