push0: Scalable and Fault-Tolerant Orchestration for Zero-Knowledge Proof Generation

Abstract

Zero-knowledge proof generation imposes stringent timing and reliability constraints on blockchain systems. For ZK-rollups, delayed proofs cause finality lag and economic loss; for Ethereum's emerging L1 zkEVM, proofs must complete within the 12-second slot window to enable stateless validation. The Ethereum Foundation's Ethproofs initiative coordinates multiple independent zkVMs across proving clusters to achieve real-time block proving, yet no principled orchestration framework addresses the joint challenges of (i) strict head-of-chain ordering, (ii) sub-slot latency bounds, (iii) fault-tolerant task reassignment, and (iv) prover-agnostic workflow composition. We present push0, a cloud-native proof orchestration system that decouples prover binaries from scheduling infrastructure. push0 employs an event-driven dispatcher--collector architecture over persistent priority queues, enforcing block-sequential proving while exploiting intra-block parallelism. We formalize requirements drawn from production ZK-rollup operations and the Ethereum real-time proving specification, then demonstrate via production Kubernetes cluster experiments that push0 achieves 5 ms median orchestration overhead with 99--100% scaling efficiency at 32 dispatchers for realistic workloads--overhead negligible (less than 0.1%) relative to typical proof computation times of 7+ seconds. Controlled Docker experiments validate these results, showing comparable performance (3--10 ms P50) when network variance is eliminated. Production deployment on the Zircuit zkrollup (14+ million mainnet blocks since March 2025) provides ecological validity for these controlled experiments. Our design enables seamless integration of heterogeneous zkVMs, supports automatic task recovery via message persistence, and provides the scheduling primitives necessary for both centralized rollup operators and decentralized multi-prover networks.

Figures (7)

• Figure 1: push0 architecture: message bus coordinates dispatchers (which invoke provers) and collectors (which aggregate via pluggable strategies). The strategy is internal to the collector.
• Figure 2: Production ZK-rollup pipeline. Chunks are fixed-size block ranges proven independently; batches group chunks for recursive aggregation; bundles group batches for L1 submission. Proposer flow commits to these boundaries using metadata only; proving flow generates proofs in parallel. Aggregators synchronize both flows before downstream proving.
• Figure 3: Pipeline patterns. Dispatchers ($D$, green) invoke provers; Collectors ($C$, orange) aggregate. (a) Sequential: $D_1 \to Q_2 \to D_2$. (b) Parallel: multiple dispatchers share input queue. (c) Fan-in: collector waits for $k$ messages before emitting one. (d) Multi-queue: collector reads from multiple queues, synchronizes independent flows.
• Figure 4: Multi-queue pipeline: input $Q_1$ feeds two independent dispatcher chains. Collector $C$ reads from both $Q_3$ and $Q_5$, emitting to $Q_6$ only when matching messages from both flows arrive. Flows may complete at different rates, so the collector buffers faster results until the slower flow catches up.
• Figure 5: Orchestration overhead CDFs from controlled Docker deployment (n=10 runs per rate). Left: full range (0--30 ms) shows all rates achieve sub-20 ms P99. Right: zoomed view (0--15 ms) highlights 50 and 100 tasks/sec maintain 3--6 ms overhead. Production cluster results (not shown) exhibit similar distributions with 5.2 ms median (\ref{['tab:latency-rates']}).

Theorems & Definitions (7)

• Theorem 1: Starvation Freedom
• proof : Proof sketch
• Theorem 2: Barrier Locality
• proof
• Corollary 1: Barrier Completion
• Theorem 3: Routing Continuity
• proof