From Skew to Symmetry: Node-Interconnect Multi-Path Balancing with Execution-time Planning for Modern GPU Clusters

Abstract

Modern GPU-based high-performance computing clusters offer unprecedented communication bandwidth through heterogeneous intra-node interconnects and inter-node networks. However, despite this high aggregate bandwidth, many real-world communication patterns fail to fully utilize the available hardware. Traffic skew often leads to situations where a small subset of links becomes oversaturated while others remain underutilized, resulting in congestion, latency spikes, and poor scalability. Existing communication frameworks such as NCCL and MPI with UCX typically rely on static fastest-path routing or hashing-based multi-rail striping, which leaves significant bandwidth unused when runtime traffic deviates from expected distributions. To address these limitations, we propose NIMBLE (Node-Interconnect Multi-path Balancing with Execution-time orchestration), a runtime communication orchestration system that dynamically redistributes traffic to balance link utilization across all available intra-node and inter-node paths. NIMBLE formulates this as a capacity-normalized minimum-congestion optimization problem and solves it efficiently using a multiplicative-weights algorithm. It further employs CUDA-aware GPU kernel-based RDMA pipelining to route traffic through intermediate GPUs and rail-matched NICs. The system is endpoint-driven, integrates transparently with existing communication libraries without requiring application changes, and preserves ordering, determinism, and low overhead. On H100-SXM4 nodes with fully connected NVLink and four NDR400 rails, NIMBLE achieves up to 2.3x higher intra-node bandwidth and 3.8x higher inter-node throughput compared to single-path baselines. It outperforms NCCL and MPI by up to 5.2x on skewed All-to-Allv workloads and 1.35x on end-to-end LLM MoE workloads, while matching baseline performance under balanced traffic.

Figures (8)

• Figure 1: In skewed workload, heavy-load links (both intra-node and inter-node fabrics) dominate the overall latency. Our proposed NIMBLE is able to evaluate load imbalance and automatically redirect message transfer through less busy paths, reducing the overall maximum latency across all links.
• Figure 2: NIMBLE system architecture. Positioned between applications and hardware fabrics, NIMBLE intercepts communication calls and orchestrates dynamic link-aware data transfers across intra- and inter-node interconnects.
• Figure 3: An example of NIMBLE orchestrating communication workload from hotspot links to all available links in the system. We significantly increased the overall throughput by scattering the original point-to-point workload into an aggregation of communication paths.
• Figure 4: Given 4-GPU 4-NIC computing nodes, intra-node link balancing tries to utilize all available links that direct the path from sources GPU to destination GPU; Inter-node link balancing leverages the rail-matched path among nodes as well as intra-node links on both nodes.
• Figure 5: On every forwarding GPU, we allocate a small amount of peer-to-peer buffer and register RDMA memory region if NIC is involved in the path. These intermediate buffers are much smaller than the actual data buffer on the sender and receiver side, they serve as the buffer pool in the pipeline.