Specifying and Verifying RDMA Synchronisation (Extended Version)
Guillaume Ambal, Max Stupple, Brijesh Dongol, Azalea Raad
TL;DR
This work advances formal semantics for RDMA by introducing RDMA^{TSO}_{RMW}, the first model of remote read-modify-write operations and their weak atomicity properties. It then builds a modular framework, RDMA^{WAIT}_{RMW}, atop the LOCO/Mowgli ecosystems to specify, implement, and verify several RDMA synchronization libraries—weak, strong, and node locks—plus a strong sequentially consistent RDMA^{SC}_{RMW} library. The authors extend the mowgli framework to accommodate remote RMWs, enabling compositional verification of libraries that interact through shared variables and global fences. The approach leverages NvI-based collaboration and the LOCO sv library to enforce mutual exclusion while preserving modularity and verifiability. The resulting libraries demonstrate concrete correctness guarantees for networked synchronization primitives and highlight how per-node locking and global fencing can achieve SC-like behavior in RDMA-based systems, with broad implications for high-performance distributed runtimes and data structures.
Abstract
Remote direct memory access (RDMA) allows a machine to directly read from and write to the memory of remote machine, enabling high-throughput, low-latency data transfer. Ensuring correctness of RDMA programs has only recently become possible with the formalisation of $\text{RDMA}^\text{TSO}$ semantics (describing the behaviour of RDMA networking over a TSO CPU). However, this semantics currently lacks a formalisation of remote synchronisation, meaning that the implementations of common abstractions such as locks cannot be verified. In this paper, we close this gap by presenting $\text{RDMA}^{\text{TSO}}_{\text{RMW}}$, the first semantics for remote `read-modify-write' (RMW) instructions over TSO. It turns out that remote RMW operations are weak and only ensure atomicity against other remote RMWs. We therefore build a set of composable synchronisation abstractions starting with the $\text{RDMA}^{\text{WAIT}}_{\text{RMW}}$ library. Underpinned by $\text{RDMA}^{\text{WAIT}}_{\text{RMW}}$, we then specify, implement and verify three classes of remote locks that are suitable for different scenarios. Additionally, we develop the notion of a strong RDMA model, $\text{RDMA}^{\text{SC}}_{\text{RMW}}$, which is akin to sequential consistency in shared memory architectures. Our libraries are built to be compatible with an existing set of high-performance libraries called LOCO, which ensures compositionality and verifiability.