Reimagining RDMA Through the Lens of ML

TL;DR

Tail latency in distributed ML communications is dominated by collective synchronization across thousands of GPUs. The paper proposes Celeris, a domain-specific, stateless RDMA NIC that removes retransmissions and in-order delivery, delegating loss handling to the ML pipeline while preserving hardware congestion control. FPGA and cluster-scale simulations show improvements including up to 2.3× reduction in the 99th percentile latency, BRAM reductions of about 63.5–72.7%, and MTBF improvements to around 80.5 hours. This ML-aligned transport reduces NIC complexity, lowers hardware cost, and enables scalable, fault-tolerant cluster-scale ML.

Abstract

As distributed machine learning (ML) workloads scale to thousands of GPUs connected by ultra-high-speed inter-connects, tail latency in collective communication has emerged as a primary bottleneck. Prior RDMA designs, like RoCE, IRN, and SRNIC, enforce strict reliability and in-order delivery, relying on retransmissions and packet sequencing to ensure correctness. While effective for general-purpose workloads, these mechanisms introduce complexity and latency that scale poorly, where even rare packet losses or delays can consistently degrade system performance. We introduce Celeris, a domain-specific RDMA transport that revisits traditional reliability guarantees based on ML's tolerance for lost or partial data. Celeris removes retransmissions and in-order delivery from the RDMA NIC, enabling best-effort transport that exploits the robustness of ML workloads. It retains congestion control (e.g., DCQCN) and manages communication with software-level mechanisms such as adaptive timeouts and data prioritization, while shifting loss recovery to the ML pipeline (e.g., using the Hadamard Transform). Early results show that Celeris reduces 99th-percentile latency by up to 2.3x, cuts BRAM usage by 67%, and nearly doubles NIC resilience to faults -- delivering a resilient, scalable transport tailored for ML at cluster scale.

Figures (2)

• Figure 1: Training and inference accuracy of all models remains stable under partial network drops ($\leq$ 5%).
• Figure 2: AllReduce step times under background contention. Celeris bounds tail latency with per-step timeouts.