HybridEP: Scaling Expert Parallelism to Cross-Datacenter Scenario via Hybrid Expert/Data Transmission

TL;DR

HybridEP tackles the cross-datacenter MoE training bottleneck by addressing Expert Parallelism (EP) under constrained bandwidth. It introduces a stream-based modeling framework that decouples computation and communication to predict end-to-end latency and derive an optimal data-expert hybrid transmission ratio. The design combines Domain-Based Partition and Parameter-Efficient Migration (with SR-based compression and asynchronous communication) to realize the modeled optimum in practice. Across real clusters and large-scale simulations, HybridEP achieves up to 5.68× speedup over state-of-the-art MoE systems and up to 1.45× speedup with 1000 DCs across bandwidths.

Abstract

Mixture-of-Experts (MoE) has become a popular architecture for scaling large models. However, the rapidly growing scale outpaces model training on a single DC, driving a shift toward a more flexible, cross-DC training paradigm. Under this, Expert Parallelism (EP) of MoE faces significant scalability issues due to the limited cross-DC bandwidth. Specifically, existing EP optimizations attempt to overlap data communication and computation, which has little benefit in low-bandwidth scenarios due to a much longer data communication time. Therefore, the trends of cross-DC EP scaling is fast becoming a critical roadblock to the continued growth of MoE models. To address this, we propose HybridEP, a modeling-guided framework to optimize EP under constrained bandwidth. Our key idea is to dynamically transform the spatial placement of experts to reduce data communication traffic and frequency, thereby minimizing EP's communication overheads. However, it is non-trivial to find the optimal solution because it complicates the original communication pattern by mixing data and expert communication. We therefore build a stream-based model to determine the optimal transmission ratio. Guided by this, we incorporate two techniques: (1) domain-based partition to construct the mapping between hybrid patterns and specific communication topology at GPU level, and (2) parameter-efficient migration to further refine this topology by reducing expert transmission overhead and enlarging the domain size. Combining all these designs, HybridEP can be considered as a more general EP with better scalability. Experimental results show that HybridEP outperforms existing state-of-the-art MoE training systems by up to 5.6x under constrained bandwidth. We further compare HybridEP and EP on large-scale simulations. HybridEP achieves up to 1.45x speedup with 1k DCs under different bandwidths.

Figures (17)

• Figure 1: Structures of transformer and MoE block under real training configuration (2 nodes $\times$ 2 GPUs).Each MoE block replicates FFN to multiple experts, and only activates part of them (through gate network) to process data with constant computation cost. However, the frequent communication of All-to-All makes it the main bottleneck during training.
• Figure 2: Analysis of parallelisms under common settings and the scalability of EP.(a) shows the trends of more DCs and larger EP, which requires more interconnections between DCs and scaling EP across DCs (b) shows the overhead ratio of EP under different bandwidths. This trend reveals a heavy overhead caused by EP when scaling under low bandwidth.
• Figure 3: An example of how the spatial placement of data and experts affect EP's efficiency.(a) shows the original placement of EP, where 7 low-bandwidth communications exist with slowest training speed. (b) shows the ideal case without any low-bandwidth communication through reshaping placements of data, which achieves the best efficiency. (c) shows a possible optimization in practice with only 4 low-bandwidth communications, achieved only by reshaping the placement of experts. This achieves a good compromise between ideal case and original EP.
• Figure 4: Compressibility analysis of data and experts.The distribution of data has large magnitude and many outliers (red part), while the distribution of expert weight is relative small and flat, leading to a higher compressibility.
• Figure 5: The modeling of training process and the communication breakdown of A2A and AG.(a) A shows a modeling process using the divide-and-conquer approach. The training process is first split into independent modeling of computing and communication streams, and then their overlapping relationships are considered for merging. (b) shows that the traffic of A2A remains unchanged (i.e., O(1)), while the traffic of AG is multiplied by number of GPUs (i.e., O(n)).