UCCL-EP: Portable Expert-Parallel Communication

TL;DR

MoE-based large language models rely on expert-parallelism (EP), but GPU-initiated token-level EP systems like DeepEP suffer from poor portability across diverse GPUs and NICs due to tight GPU-NIC coupling. UCCL-EP introduces a portable architecture that decouples initiation from execution by using a high-throughput CPU proxy layer and a lightweight 128-bit TransferCmd channel, enabling GPUDirect RDMA while enforcing NIC-specific delivery semantics via immediate data. The approach delivers DeepEP-level performance on heterogeneous platforms (e.g., NVIDIA+AWS EFA) and maintains competitive results on NVIDIA-only setups, with additional gains in SGLang inference and Megatron-LM training on AMD+Broadcom configurations. This design reduces vendor lock-in and lowers porting cost to new GPU/NIC combinations, offering a practical path to scalable EP across diverse accelerators and networks.

Abstract

Mixture-of-Experts (MoE) workloads rely on expert parallelism (EP) to achieve high GPU efficiency. State-of-the-art EP communication systems such as DeepEP demonstrate strong performance but exhibit poor portability across heterogeneous GPU and NIC platforms. The poor portability is rooted in architecture: GPU-initiated token-level RDMA communication requires tight vertical integration between GPUs and NICs, e.g., GPU writes to NIC driver/MMIO interfaces. We present UCCL-EP, a portable EP communication system that delivers DeepEP-level performance across heterogeneous GPU and NIC hardware. UCCL-EP replaces GPU-initiated RDMA with a high-throughput GPU-CPU control channel: compact token-routing commands are transferred to multithreaded CPU proxies, which then issue GPUDirect RDMA operations on behalf of GPUs. UCCL-EP further emulates various ordering semantics required by specialized EP communication modes using RDMA immediate data, enabling correctness on NICs that lack such ordering, e.g., AWS EFA. We implement UCCL-EP on NVIDIA and AMD GPUs with EFA and Broadcom NICs. On EFA, it outperforms the best existing EP solution by up to $2.1\times$ for dispatch and combine throughput. On NVIDIA-only platform, UCCL-EP achieves comparable performance to the original DeepEP. UCCL-EP also improves token throughput on SGLang by up to 40% on the NVIDIA+EFA platform, and improves DeepSeek-V3 training throughput over the AMD Primus/Megatron-LM framework by up to 45% on a 16-node AMD+Broadcom platform.

Figures (17)

• Figure 1: Assuming $m$ GPU vendors and $n$ NIC vendors, UCCL-EP enables $O(m)$ effort, instead of IBGDA's $O(m\times n)$, to support GPU-initiated token-level communication for expert parallelism.
• Figure 2: The MoE communication pattern consists of the dispatch phase and the combine phase. In the dispatch phase, the router sends each token’s input activations to one or more selected experts. In the combine phase, expert activations are collected and aggregated back to the sender. The dispatch and combine phases feature irregular, fine-grained communication.
• Figure 3: GPU-initiated token-level communication in DeepEP (High-throughput mode, single batch for illustration). $C$ stands for intra-node data copying via NVLink, $P$ stands for processing, and $R$ stands for RDMA communication. Different phases (computation, communication, and copying) interleave.
• Figure 4: GPU-initiated token-level communication outperforms coarse-grained bulk transfer (e.g., packing tokens into a contiguous buffer on GPU then CPU initiating a single contiguous transfer) on NV_EFA3 (testbed details listed in Table \ref{['tab:eval_testbed']}). The y-axis is in log scale.
• Figure 5: UCCL-EP architecture. Control buffer temporarily buffers control messages (e.g., atomics) until the conditional check specified by the message is passed, upon which the values carried by these control messages are applied. Multiple communication channels are used for moving the data payloads (e.g., tokens) via GPUDirect RDMA.