SDR-RDMA: Software-Defined Reliability Architecture for Planetary Scale RDMA Communication

TL;DR

SDR-RDMA tackles the reliability bottleneck in inter-datacenter RDMA by introducing a software-defined reliability stack that decouples application-specific reliability logic from NIC progress engines. Central to SDR is a partial completion bitmap API that exposes per-chunk reception, enabling SR and EC schemes to be implemented on commodity NICs and offloaded to the Data Path Accelerator for line-rate performance. The work provides a complete architecture, a protocol design, a mathematical performance model, and extensive evaluation—including simulations and a 400 Gbit/s real hardware testbed—showing that EC can outperform SR in certain cross-DC regimes while SR remains favorable at very low drop rates or very large messages. The approach enables rapid experimentation and deployment of reliability strategies without waiting for new NIC silicon, significantly advancing inter-datacenter GPU training by improving reliability and tail latency across diverse network conditions.

Abstract

RDMA is vital for efficient distributed training across datacenters, but millisecond-scale latencies complicate the design of its reliability layer. We show that depending on long-haul link characteristics, such as drop rate, distance and bandwidth, the widely used Selective Repeat algorithm can be inefficient, warranting alternatives like Erasure Coding. To enable such alternatives on existing hardware, we propose SDR-RDMA, a software-defined reliability stack for RDMA. Its core is a lightweight SDR SDK that extends standard point-to-point RDMA semantics -- fundamental to AI networking stacks -- with a receive buffer bitmap. SDR bitmap enables partial message completion to let applications implement custom reliability schemes tailored to specific deployments, while preserving zero-copy RDMA benefits. By offloading the SDR backend to NVIDIA's Data Path Accelerator (DPA), we achieve line-rate performance, enabling efficient inter-datacenter communication and advancing reliability innovation for inter-datacenter training.

Figures (16)

• Figure 1: RDMA stack based on the software-defined reliability (SDR) architecture. In all figures throughout the paper, green corresponds to the hardware-related parts of the SDR stack and blue encodes software components.
• Figure 2: Packet drop rate measured with iperf3 between 16 UDP flows located in Lugano and Lausanne CSCS datacenter sites approximately 350 km apart, connected via a 100 Gbit/s channel. Copper cables are used within both datacenter sites. An optical connection provided by a local ISP is used to connect the two datacenters. For each payload size, drop rates are measured over 200 trials of 15 seconds each, conducted over a 3-day period.
• Figure 3: Impact of reliability on message time at 400 Gbit/s. Left (a)3750 km $=25$ ms RTT, $P_{drop}=10^{-5}$Middle (b)8 GiB message, $P_{drop}=10^{-5}$Right (c)128 MiB message, 3750 km
• Figure 4: Example of SDR one-shot send data path with four packets, two packets per receive bitmap chunk. The protocol backend can be offloaded to the NIC.
• Figure 5: Sender and receiver SDR QPs share a zero-based root memory key, enabling offset-based addressing of receive buffers. For a QP with maximum message size $M$, message $i$ targets the offset range $[i \cdot M, i \cdot M + M)$.