Reliable and Resilient Collective Communication Library for LLM Training and Serving

TL;DR

R$^2$CCL tackles the problem of network fragility in tens-to-thousands of GPU clusters by providing a fault-tolerant collective communication library that performs hot repair through multi-NIC live migration, bilateral failure awareness, and failure-aware scheduling. It introduces two key live-migration techniques (GPU-NIC multi-registration and DMA buffer rollback) and two scheduling strategies (R$^2$CCL-Balance and R$^2$CCL-AllReduce) to handle single and multiple NIC failures, including recursive scheduling for multi-failure scenarios. Empirical results on 2-node testbeds and large-scale simulations show overheads of less than $1 ext{--}3 ext{ } ext{%}$ for training and inference under failures, with orders-of-magnitude improvements over AdapCC and DéjàVu in their respective domains. The work demonstrates that network redundancy and intelligent scheduling can maintain throughput and latency targets in fault-prone environments, reducing downtime and cost for both training and serving of large models.

Abstract

Modern ML training and inference now span tens to tens of thousands of GPUs, where network faults can waste 10--15\% of GPU hours due to slow recovery. Common network errors and link fluctuations trigger timeouts that often terminate entire jobs, forcing expensive checkpoint rollback during training and request reprocessing during inference. We present R$^2$CCL, a fault-tolerant communication library that provides lossless, low-overhead failover by exploiting multi-NIC hardware. R$^2$CCL performs rapid connection migration, bandwidth-aware load redistribution, and resilient collective algorithms to maintain progress under failures. We evaluate R$^2$CCL on two 8-GPU H100 InfiniBand servers and via large-scale ML simulators modeling hundreds of GPUs with diverse failure patterns. Experiments show that R$^2$CCL is highly robust to NIC failures, incurring less than 1\% training and less than 3\% inference overheads. R$^2$CCL outperforms baselines AdapCC and DejaVu by 12.18$\times$ and 47$\times$, respectively.

Figures (16)

• Figure 1: R$^2$CCL vs. Checkpoint.
• Figure 2: R$^2$CCL Overview.
• Figure 3: R$^2$CCL-HotRepair vs. R$^2$CCL-Balance.
• Figure 4: ReduceScatter, AllGather and AllReduce.
• Figure 5: R$^2$CCL-AllReduce reduce bottleneck server workload from 2D to 1.75D based on the lost bandwidth.Total data size D with 4 chunks, ring Allreduce each worker need to send and receive nearly 2D size. Broadcast need to send and receive D size. R$^2$CCL decompose it to allreduce and broadcast combination according the bandwidth ratio to reduce the completion time.