Pilotfish: Distributed Execution for Scalable Blockchains

TL;DR

Pilotfish presents a scalable, crash-tolerant, scale-out execution framework for blockchains by distributing execution across intravalidator workers. It combines SequencingWorkers for data gathering with ExecutionWorkers that own state shards, enabling horizontal scalability while preserving serializability and determinism through a novel versioned-queue scheduler. The system achieves linear throughput growth up to at least $8$ workers per validator on compute-bound workloads, while maintaining low latency, and provides robust recovery via reconfiguration and checkpoint synchronization. This work demonstrates a practical path to elastic blockchain execution, reducing over-provisioning and unlocking higher-throughput, adaptable blockchain infrastructures.

Abstract

Scalability is a crucial requirement for modern large-scale systems, enabling elasticity and ensuring responsiveness under varying load. While cloud systems have achieved scalable architectures, blockchain systems remain constrained by the need to over-provision validator machines to handle peak load. This leads to resource inefficiency, poor cost scaling, and limits on performance. To address these challenges, we introduce Pilotfish, the first scale-out transaction execution engine for blockchains. Pilotfish enables validators to scale horizontally by distributing transaction execution across multiple worker machines, allowing elasticity without compromising consistency or determinism. It integrates seamlessly with the lazy blockchain architecture, completing the missing piece of execution elasticity. To achieve this, Pilotfish tackles several key challenges: ensuring scalable and strongly consistent distributed transactions, handling partial crash recovery with lightweight replication, and maintaining concurrency with a novel versioned-queue scheduling algorithm. Our evaluation shows that Pilotfish scales linearly up to at least eight workers per validator for compute-bound workloads, while maintaining low latency. By solving scalable execution, Pilotfish brings blockchains closer to achieving end-to-end elasticity, unlocking new possibilities for efficient and adaptable blockchain systems.

Figures (14)

• Figure 1: Pilotfish validator's components. Each validator is composed of several SequencingWorkers to fetch and persist the client's transaction, one Primary to run Byzantine agreement on metadata, and several ExecutionWorkers to execute transactions. Each component may run on dedicated machines or be collocated with other components. Dotted arrows indicate internal messages exchanged between the components of the validator (localhost or LAN) and solid arrows indicate messages exchanged with the outside world (WAN).
• Figure 2: Pilotfish overview. Every validator runs with 5 machines: one machine running the Primary and 4 machines running workers. Each worker machine collocates 1 SequencingWorker and 1 ExecutionWorker. The Primary runs a Byzantine agreement protocol to sequence batch digests (➊). SequencingWorkers receive the committed sequence and load the data of the corresponding transactions from their storage (➋). Each ExecutionWorkers receiving these transactions assigns a lock to each object referenced by the transaction to schedule their execution (➌). A deterministically-selected ExecutionWorker eventually receives the object's data referenced by the execution and executes it (➍). Finally, the ExecutionWorker signals all SequencingWorkers to update their state with the results of the transaction's execution (➎).
• Figure 3: Example snapshot of the $\textsc{Pending}\xspace$ queues of an ExecutionWorker. Pilotfish schedules the execution of the sequence $[\textsf{Tx}\xspace_1, \textsf{Tx}\xspace_2, \textsf{Tx}\xspace_3, \textsf{Tx}\xspace_4, \textsf{Tx}\xspace_5]$. The ExecutionWorker stores $\textsf{Tx}\xspace_1$ as $(\textsf{W}, [\textsf{Tx}\xspace_1])$ in the queue of $oid\xspace_1$ as it only mutates $oid\xspace_1$. $\textsf{Tx}\xspace_2$ then mutates $oid\xspace_1$ and writes $oid\xspace_3$; it is thus store in the queue of $oid\xspace_1$ (implicitly taking $\textsf{Tx}\xspace_1$ as dependency) and $oid\xspace_3$. $\textsf{Tx}\xspace_3$ schedules a read for both $oid\xspace_1$ and $oid\xspace_2$ and a write for $oid\xspace_4$. $\textsf{Tx}\xspace_4$ reads $oid\xspace_2$ (it can thus read $oid\xspace_2$ in parallel with $\textsf{Tx}\xspace_3$, registering $(\textsf{R}, [\textsf{Tx}\xspace_3, \textsf{Tx}\xspace_4])$ in the queue of $oid\xspace_2$) and $oid\xspace_3$, and writes $oid\xspace_4$. Finally $\textsf{Tx}\xspace_5$ reads $oid\xspace_1$ (it can thus read $oid\xspace_1$ in parallel with $\textsf{Tx}\xspace_3$), writes $oid\xspace_2$ and mutates $oid\xspace_3$.
• Figure 4: Replication scheme for ExecutionWorkers. The object store is partitioned into shards, and each shard is replicated $n_e$-fold. Each row represents a cluster, and ExecutionWorkers within a cluster coordinate to process transactions. During normal operation, the only communication between clusters is the sending of checkpoints.
• Figure 5: Pilotfish latency vs throughput (a) and scalability (b) with simple transfers.

Theorems & Definitions (69)

• definition thmcounterdefinition: Pilotfish Serializability
• definition thmcounterdefinition: Pilotfish Determinism
• definition thmcounterdefinition: Pilotfish Liveness
• definition thmcounterdefinition: Sequential Schedule
• definition thmcounterdefinition: Conflicting Transactions
• lemma thmcounterlemma: Sequential Batch Processing
• proof
• lemma thmcounterlemma: Transactions Order in Queues
• proof
• lemma thmcounterlemma: Unlock after Access