Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ALock: Asymmetric Lock Primitive for RDMA Systems

Amanda Baran, Jacob Nelson-Slivon, Lewis Tseng, Roberto Palmieri

TL;DR

RDMA enables remote memory access but lacks atomicity between local and remote operations, making synchronization challenging and loopback/RPC approaches costly. ALock introduces a two-cohort mutual exclusion primitive that embeds MCS locks for local and remote accesses into a modified Peterson's algorithm, enabling loopback-free and RPC-free synchronization. The design provides fairness via a budget mechanism and reduces RDMA congestion, achieving up to 29x throughput gains and up to 20x latency reductions in local-heavy workloads, as demonstrated on a multi-node cluster. This work offers a practical, scalable synchronization primitive for RDMA systems and is released as open source with a formal correctness specification.

Abstract

Remote direct memory access (RDMA) networks are being rapidly adopted into industry for their high speed, low latency, and reduced CPU overheads compared to traditional kernel-based TCP/IP networks. RDMA enables threads to access remote memory without interacting with another process. However, atomicity between local accesses and remote accesses is not guaranteed by the technology, hence complicating synchronization significantly. The current solution is to require threads wanting to access local memory in an RDMA-accessible region to pass through the RDMA card using a mechanism known as loopback, but this can quickly degrade performance. In this paper, we introduce ALock, a novel locking primitive designed for RDMA-based systems. ALock allows programmers to synchronize local and remote accesses without using loopback or remote procedure calls (RPCs). We draw inspiration from the classic Peterson's algorithm to create a hierarchical design that includes embedded MCS locks for two cohorts, remote and local. To evaluate the ALock we implement a distributed lock table, measuring throughput and latency in various cluster configurations and workloads. In workloads with a majority of local operations, the ALock outperforms competitors up to 29x and achieves a latency up to 20x faster.

ALock: Asymmetric Lock Primitive for RDMA Systems

TL;DR

RDMA enables remote memory access but lacks atomicity between local and remote operations, making synchronization challenging and loopback/RPC approaches costly. ALock introduces a two-cohort mutual exclusion primitive that embeds MCS locks for local and remote accesses into a modified Peterson's algorithm, enabling loopback-free and RPC-free synchronization. The design provides fairness via a budget mechanism and reduces RDMA congestion, achieving up to 29x throughput gains and up to 20x latency reductions in local-heavy workloads, as demonstrated on a multi-node cluster. This work offers a practical, scalable synchronization primitive for RDMA systems and is released as open source with a formal correctness specification.

Abstract

Remote direct memory access (RDMA) networks are being rapidly adopted into industry for their high speed, low latency, and reduced CPU overheads compared to traditional kernel-based TCP/IP networks. RDMA enables threads to access remote memory without interacting with another process. However, atomicity between local accesses and remote accesses is not guaranteed by the technology, hence complicating synchronization significantly. The current solution is to require threads wanting to access local memory in an RDMA-accessible region to pass through the RDMA card using a mechanism known as loopback, but this can quickly degrade performance. In this paper, we introduce ALock, a novel locking primitive designed for RDMA-based systems. ALock allows programmers to synchronize local and remote accesses without using loopback or remote procedure calls (RPCs). We draw inspiration from the classic Peterson's algorithm to create a hierarchical design that includes embedded MCS locks for two cohorts, remote and local. To evaluate the ALock we implement a distributed lock table, measuring throughput and latency in various cluster configurations and workloads. In workloads with a majority of local operations, the ALock outperforms competitors up to 29x and achieves a latency up to 20x faster.
Paper Structure (16 sections, 6 figures, 1 table, 4 algorithms)

This paper contains 16 sections, 6 figures, 1 table, 4 algorithms.

Table of Contents

  1. Introduction
  2. RDMA Scalability and pitfalls
  3. Background
  4. Peterson's Algorithm
  5. MCS Lock
  6. System Model
  7. Alock Algorithm and Design
  8. ALock's MCS lock
  9. ALock's Peterson's Algorithm
  10. Evaluation
  11. Choosing a Budget
  12. Throughput Evaluation
  13. Latency Evaluation
  14. Related Work
  15. Conclusion
  16. ...and 1 more sections

Figures (6)

  • Figure 1: RDMA Spinlock with 1k locks on 1 Node.
  • Figure 2: Execution of the ALock algorithm in a system with two nodes, one lock per node, and one thread per node.
  • Figure 3: 64B-aligned ALock containing 8B pointers to the remote and local cohort tails, and an integer victim field to indicate the current victim cohort. Values are padded to the address alignments shown.
  • Figure 4: Relative speedup compared to the baseline remote budget of 5.
  • Figure 5: Throughput for a variety of workloads on 5, 10, and 20 nodes. The X-axis is the number of threads per node.
  • ...and 1 more figures

Theorems & Definitions (2)

  • definition 1: Local Access
  • definition 2: Remote Access