Collective Communication for 100k+ GPUs

TL;DR

NCCLX addresses the core bottlenecks of collective communication for large-scale LLM training and low-latency inference by introducing a unified, host-driven, zero-copy transport stack (CTran) built on NCCL. It combines host-centric algorithm scheduling, GPU-buffer zero-copy transfers, and topology-aware load balancing (DQPLB) with fault-tolerant mechanisms (FTAR) and GPU-resident inference primitives (AllToAllvDynamic) to scale to extremely large clusters. The key contributions include host-driven customization, zero-copy and SM-free transfers, DQPLB, FTAR, topology-aware optimizations, scalable initialization tooling, fault localization, and performance observability, together enabling efficient training and inference on clusters of well over one hundred thousand GPUs. Empirical results on Llama4 show substantial improvements in startup time (up to elevenfold), per-step training latency reductions (up to twelve percent), and end-to-end inference latency reductions (from fifteen to eighty percent) across diverse configurations, demonstrating practical impact for future ultra-scale AI deployments.

Abstract

The increasing scale of large language models (LLMs) necessitates highly efficient collective communication frameworks, particularly as training workloads extend to hundreds of thousands of GPUs. Traditional communication methods face significant throughput and latency limitations at this scale, hindering both the development and deployment of state-of-the-art models. This paper presents the NCCLX collective communication framework, developed at Meta, engineered to optimize performance across the full LLM lifecycle, from the synchronous demands of large-scale training to the low-latency requirements of inference. The framework is designed to support complex workloads on clusters exceeding 100,000 GPUs, ensuring reliable, high-throughput, and low-latency data exchange. Empirical evaluation on the Llama4 model demonstrates substantial improvements in communication efficiency. This research contributes a robust solution for enabling the next generation of LLMs to operate at unprecedented scales.

Figures (25)

• Figure 1: Multi-building network architecture.
• Figure 2: NCCLX communication stack overview. NCCLX provides three categories of APIs: Host-initiated APIs, Host-initiated APIs with GPU-resident metadata, and Device-initiated APIs.
• Figure 3: Coordination between CTran internal CPU thread and the CUDA kernel for a NCCL collective. Local D2D copy (i.e., for out-of-place collective) and P2P copy to NVLink peers are handled within ncclKernel, while RDMA network transfer is driven by the CPU thread. NCCL Kernel 1 demonstrates the fully-host-driven mode; NCCL Kernels 2 and 3 demonstrate the host-kernel coordinated mode.
• Figure 4: Data transfer breakdown from user send buffer to receive buffer via network transfer. Internal FIFO buffer is required in copy-based transfer. In addition to the required network transfer from sender side NIC to receiver side NIC, (a) copy-based transfer includes: (1) D2D copy from send buffer to sender side FIFO, (2) PCIe transfer from sender side FIFO to NIC, (3) PCIe transfer from NIC to receiver side FIFO, and (4) D2D copy from receiver side FIFO to receive buffer . In contrast, (b) includes: (1) PCIe transfer from send buffer to sender side NIC, and (2) PCIe transfer from receiver side NIC to receiver buffer.
• Figure 5: Copy + RDMA pipeline in copy-based transfer between two GPUs. Both sender and receiver employ NCHANNELS number of pipelines, each called channel. Each channel is driven by a dedicated GPU thread block. Each channel further forms a NSTEPS-way pipeline via two multi-slot FIFO buffers on sender and receiver, respectively. Sender side pipeline involves device-to-device copy from user send buffer (sbuf) to FIFO, followed by RDMA from the FIFO slot; receiver side pipeline starts with RDMA receive for a FIFO slot, followed by a device-to-device copy to the destination receive buffer (rbuf).