Space-Optimal, Computation-Optimal, Topology-Agnostic, Throughput-Scalable Causal Delivery through Hybrid Buffering

TL;DR

The paper tackles topology-agnostic causal delivery in large distributed systems, where metadata and processing costs of receiver-buffering become prohibitive. It introduces Sender Permission to Send (SPS) as a foundational enforcement principle and analyzes existing sender-buffering approaches, notably Cykas, identifying liveness and throughput limitations. The authors then propose a novel hybrid-buffering algorithm that combines sender-side SPS with receiver-side buffering, achieving effectively constant per-message metadata ($O(1)$) and amortized constant-time computation, while remaining topology-agnostic and throughput-scalable. Although not latency-optimal, the approach provides a practical, scalable solution for systems with thousands of processes and can serve as a building block for topology-aware hybrids that trade throughput for lower latency. The work lays groundwork for future refinements, including extended receiver-buffering to further reduce latency under controlled metadata costs.

Abstract

Message delivery respecting causal ordering (causal delivery) is one of the most classic and widely useful abstraction for inter-process communication in a distributed system. Most approaches tag messages with causality information and buffer them at the receiver until they can be safely delivered. Except for specific approaches that exploit communication topology, therefore not generally applicable, they incur a metadata overhead which is prohibitive for a large number of processes. Much less used are the approaches that enforce causal order by buffering messages at the sender, until it is safe to release them to the network, as the classic algorithm has too many drawbacks. In this paper, first we discuss the limitations of sender-only buffering approaches and introduce the Sender Permission to Send (SPS) enforcement strategy, showing that SPS + FIFO implies Causal. We analyze a recent sender-buffering algorithm, Cykas, which follows SPS + FIFO, albeit very conservatively, pointing out throughput scalability and liveness issues. Then, we introduce a novel SPS + FIFO based algorithm, which adopts a new hybrid approach: enforcing causality by combining sender-buffering to enforce SPS and receiver-buffering to enforce FIFO. The algorithm overcomes limitations of sender-only buffering, and achieves effectively constant metadata size per message. By a careful choice of data-structures, the algorithm is also computationally-optimal, with amortized effectively constant processing overhead. As far as we know, there is no other topology-agnostic causal delivery algorithm with these properties.

Figures (7)

• Figure 1: Happens-before between application messages. Some process receives (r), buffers, and then delivers (d) message $a$, and similarly for message $b$. The process causal-sends (c) message $c$, keeps it in the send-buffer, and then network-sends (s) it. We have $a \mathrel{\hbox{$\mathsf{hb}$}} c$ but $b \not\mathrel{\hbox{$\mathsf{hb}$}} c$, because, even if $b$ is received before, it is not delivered before $c$ is causal-sent. It does not matter that $b$ is delivered before $c$ is network-sent.
• Figure 2: Spacetime diagram. Process $i$ causal-sends message $m$ after receiving and delivering message $a$ (from process $j$) and $b$ (from process $k$). Message $m$ is kept in the send-buffer until it can be network-sent, when $i$ knows that all causal predecessors sent by either $j$ or $k$ have been delivered. Optimal time in the buffer shown in green; time taken by Cykas shown in red.
• Figure 3: Sliding array and sliding map (combination of sliding bit-array and map).
• Figure 4: Basic causal delivery algorithm for process $i$.
• Figure 5: Example showing unified buffer and missing permits. All messages before index $m1$ have been network-sent and no message starting from index $m2$ have been network-sent. Some messages between $m1$ and $m2$ may have been network-sent and even acked ($\bot$ payload). At index $u2$ is the first unacked message to a different receiver than the oldest message in the buffer ($f.\mathord{\hbox{$\mathsf{rcv}$}} \neq c.\mathord{\hbox{$\mathsf{rcv}$}} = e.\mathord{\hbox{$\mathsf{rcv}$}}$). Permit at index $p2$ is the first from a different sender ($s.\mathord{\hbox{$\mathsf{snd}$}} \neq p.\mathord{\hbox{$\mathsf{snd}$}} = r.\mathord{\hbox{$\mathsf{snd}$}}$). Message at index $m2$ is the first depending on the permit at $p2$ ($l.\mathord{\hbox{$\mathsf{perm}$}} > p2$ and $k.\mathord{\hbox{$\mathsf{perm}$}} \leq p2$).

Theorems & Definitions (16)

• Definition 4.1: Sender Permission to Send (SPS) enforcement strategy
• Proposition 4.2
• proof
• Definition 4.3: Conservative Sender Permission to Send (CSPS) enforcement strategy
• Proposition 4.4
• proof
• Proposition 5.1
• proof
• Proposition 5.2
• proof