Ira: Efficient Transaction Replay for Distributed Systems

TL;DR

This work presents Ira, a framework to accelerate backup replay by transmitting compact emph{hints} alongside transaction batches, and presents a concrete protocol, Ira-L, within this framework to improve cache management of Ethereum block execution.

Abstract

In primary-backup replication, consensus latency is bounded by the time for backup nodes to replay (re-execute) transactions proposed by the primary. In this work, we present Ira, a framework to accelerate backup replay by transmitting compact \emph{hints} alongside transaction batches. Our key insight is that the primary, having already executed transactions, possesses knowledge of future access patterns which is exactly the information needed for optimal replay. We use Ethereum for our case study and present a concrete protocol, Ira-L, within our framework to improve cache management of Ethereum block execution. The primaries implementing Ira-L provide hints that consist of the working set of keys used in an Ethereum block and one byte of metadata per key indicating the table to read from, and backups use these hints for efficient block replay. We evaluated Ira-L against the state-of-the-art Ethereum client reth over two weeks of Ethereum mainnet activity ($100,800$ blocks containing over $24$ million transactions). Our hints are compact, adding a median of $47$ KB compressed per block ($\sim5\%$ of block payload). We observe that the sequential hint generation and block execution imposes a $28.6\%$ wall-time overhead on the primary, though the direct cost from hints is $10.9\%$ of execution time; all of which can be pipelined and parallelized in production deployments. On the backup side, we observe that Ira-L achieves a median per-block speedup of $25\times$ over baseline reth. With $16$ prefetch threads, aggregate replay time drops from $6.5$ hours to $16$ minutes ($23.6\times$ wall-time speedup).

Figures (18)

• Figure 1: LRU vs. Belady-optimal caching (cache size is $2$). For access sequence $C$, $A$, $C$ starting with cache $\{A, B\}$: LRU evicts $A$ (least recently used) when $C$ arrives, then misses when it accesses $A$ next. Optimal evicts $B$ (never accessed again), achieving one fewer miss. With hints, backups can make optimal eviction decisions.
• Figure 2: Ira architecture. The primary executes transactions while building a hint that records access information. Backups use these hints to implement optimal cache eviction during replay.
• Figure 3: Execution time breakdown showing I/O dominance. The stacked area shows I/O time (blue) and compute time (yellow) per block; the line shows transaction count. I/O consistently dominates regardless of block size.
• Figure 4: Per-block operation distribution showing storage dominance. Storage operations (blue) consistently account for the majority of state accesses across all blocks.
• Figure 5: Intra-block access frequency distribution. Most keys are accessed 1--2 times per block, but a long tail of frequently-accessed keys drives high locality.