MixServe: An Automatic Distributed Serving System for MoE Models with Hybrid Parallelism Based on Fused Communication Algorithm

TL;DR

MixServe tackles the inter-node communication bottlenecks in distributed MoE model serving by introducing a TP-EP hybrid parallelism guided by a fused AR-A2A communication scheme. The system automatically analyzes model hyperparameters and hardware/network configurations (offline) to select an optimal parallel strategy and then deploys the corresponding weight shards online, injecting mixed parallel communication into the forward path. A formal analyzer defines parallel strategies, analyzes AR/A2A operators, and balances DP and EP, while fused AR-A2A algorithms overlap intra-node and inter-node communication to reduce latency. Evaluations on DeepSeek-R1 and Qwen3 show up to 3.80x TTFT improvement, up to 1.66x ITL improvement, and up to 50.3% throughput gains, demonstrating practical scalability for real-world MoE serving. The work provides a principled, theory-backed, practical path to efficient serving of trillion-parameter MoE models in heterogeneous distributed environments.

Abstract

The Mixture of Experts (MoE) models are emerging as the latest paradigm for Large Language Models (LLMs). However, due to memory constraints, MoE models with billions or even trillions of parameters can only be deployed in multi-GPU or even multi-node & multi-GPU based serving systems. Thus, communication has became a major bottleneck in distributed serving systems, especially inter-node communication. Contemporary distributed MoE models are primarily implemented using all-reduce (AR) based tensor parallelism (TP) and all-to-all (A2A) based expert parallelism (EP). However, TP generally exhibits low inter-node efficiency and is thus confined to high-speed intra-node bandwidth. In contrast, EP tends to suffer from load imbalance, especially when the parallel degree is high. In this work, we introduce MixServe, a novel automatic distributed serving system for efficient deployment of MoE models by a novel TP-EP hybrid parallelism based on fused AR-A2A communication algorithm. MixServe begins by evaluating the communication overhead associated with various parallel strategies, taking into account the model hyperparameters and the configurations of network and hardware resources, and then automatically selects the most efficient parallel strategy. Then, we propose the TP-EP hybrid parallelism based on fused AR-A2A communication algorithm that overlaps intra-node AR communication and inter-node A2A communication. Extensive experiments on DeepSeek-R1 and Qwen3 models demonstrate that MixServe achieves superior inference performance, with 1.08~3.80x acceleration in time to first token (TTFT), 1.03~1.66x acceleration in inter-token latency (ITL), and 5.2%~50.3% throughput improvement compared to existing approaches.

Figures (12)

• Figure 1: Tensor Parallelism (TP) and All Reduce (AR) operators.
• Figure 2: Expert Parallelism (EP) and All To All (A2A) operators.
• Figure 3: Communication overhead of AR and A2A operators. The left subfigure shows the communication overhead for DeepSeek-R1 DeepSeek-R1 and Qwen3 Qwen3 models with different parallel degrees, while the right subfigure presents the results for intra-node and inter-node communication with different data sizes.
• Figure 4: Gantt chart of the comparison between EP and TP+EP in a single MoE block, calculated from profiling data of DeepSeek-R1 in a 4-node cluster with 8 NPUs each.
• Figure 5: MixServe system overview.