Orderly Management of Packets in RDMA by Eunomia

TL;DR

RDMA datacenters are hampered by the in-order delivery requirement, which prevents leveraging performance-enhancing techniques that rely on packet reordering. The authors present Eunomia, an on-NIC ordering layer built around a Hybrid-Dynamic Bitmap and a dynamic memory controller to track and manage reordered packets while preserving ability to deliver data in order to applications. FPGA implementation and ns-3-based simulations show that Eunomia enables fine-grained load balancing, irregular topologies like Jellyfish, and robust failure management, delivering substantial improvements in flow completion times and reductions in PFC pauses (e.g., up to 85% mean FCT reductions and ~99% PFC pause reductions in certain setups). This work demonstrates that providing ordering support in RDMA networks can unlock a wide range of performance techniques, offering significant practical impact for modern datacenter networks while maintaining a modest memory footprint.

Abstract

To fulfill the low latency requirements of today's applications, deployment of RDMA in datacenters has become prevalent over the recent years. However, the in-order delivery requirement of RDMAs prevents them from leveraging powerful techniques that help improve the performance of datacenters, ranging from fine-grained load balancers to throughput-optimal expander topologies. We demonstrate experimentally that these techniques significantly deteriorate the performance in an RDMA network because they induce packet reordering. Furthermore, lifting the in-order delivery constraint enhances the flexibility of RDMA networks and enables them to employ these performance-enhancing techniques. To realize this, we propose an ordering layer, Eunomia, to equip RDMA NICs to handle packet reordering. Eunomia employs a hybrid-dynamic bitmap structure that efficiently uses the limited on-chip memory with the help of a customized memory controller and handles high degrees of packet reordering. We evaluate the feasibility of Eunomia through an FPGA-based implementation and its performance through large-scale simulations. We show that Eunomia enables a wide range of applications in RDMA datacenter networks, such as fine-grained load balancers which improve performance by reducing average flow completion times by 85% and 52% compared to ECMP and Conweave, respectively, or employment of RDMA in expander topologies like Jellyfish which allows up to 60% lower flow completion times and higher throughput gains compared to Fat tree.

Figures (20)

• Figure 1: Fine-grained LBs significantly deteriorate in performance in (a) lossless RDMA and (b) lossy RDMA due to (c) heavy reordering, and pairing them with an ordering layer allows them to reach their full potential.
• Figure 2: Ordering support enables RDMA networks to maintain their performance under link failures in a Clos topology.
• Figure 3: Employing RDMA without ordering support in Jellyfish topology significantly degrades performance.
• Figure 4: Network factors impacting packet reordering
• Figure 5: Hybrid-Dynamic Reordering Bitmap (Linear Representation) - Each block represents a bit in the bitmap array and the number inside represents the corresponding sequence number (consecutive for convenience). Single hatched is circular bitmap while double hatched is linear. If highlighted the bit is ON, otherwise OFF.