MultiChain Blockchain Data Provenance for Deterministic Stream Processing with Kafka Streams: A Weather Data Case Study

TL;DR

This work tackles auditability and reproducibility in real-time data streams by anchoring per-window provenance on a MultiChain blockchain while keeping payloads off-chain. It combines epoch-aligned tumbling windows, canonical JSON serialization, and Merkle-tree commitments to enable deterministic replay and third-party verification, with a formal verification procedure that reconstructs on-chain anchors from off-chain data. The evaluation on Berlin weather data shows linear verification cost in window size and practical blockchain throughput, demonstrating scalable, auditable streaming analytics. Overall, the approach offers a practical external reference layer for verifiable streaming results with deterministic reprocessing capabilities.

Abstract

Auditability and reproducibility still are critical challenges for real-time data streams pipelines. Streaming engines are highly dependent on runtime scheduling, window triggers, arrival orders, and uncertainties such as network jitters. These all derive the streaming pipeline platforms to throw non-determinist outputs. In this work, we introduce a blockchain-backed provenance architecture for streaming platform (e.g Kafka Streams) the publishes cryptographic data of a windowed data stream without publishing window payloads on-chain. We used real-time weather data from weather stations in Berlin. Weather records are canonicalized, deduplicated, and aggregated per window, then serialised deterministically. Furthermore, the Merkle root of the records within the window is computed and stored alongside with Kafka offsets boundaries to MultiChain blockchain streams as checkpoints. Our design can enable an independent auditor to verify: (1) the completeness of window payloads, (2) canonical serialization, and (3) correctness of derived analytics such as minimum/maximum/average temperatures. We evaluated our system using real data stream from two weather stations (Berlin-Brandenburg and Berlin-Tempelhof) and showed linear verification cost, deterministic reproducibility, and with a scalable off-chain storage with on-chain cryptographic anchoring. We also demonstrated that the blockchain can afford to be integrated with streaming platforms particularly with our system, and we get satisfactory transactions per second values.

Figures (9)

• Figure 1: Two weather stations emitting a regional temperature to a stream processing pipeline. The processing logic builds stream window base on processing time $\leq$ ingestion time. Streaming pipeline produces two different results in each execution.
• Figure 2: Kafka Streams Architecture
• Figure 3: MultiChain Architecture
• Figure 4: Architecture overview: Weather stream sources $\rightarrow$ merge weather sources $\rightarrow$ deterministic windowing $\rightarrow$ canonicalization and Merkle commitment $\rightarrow$ checkpoint anchoring $\rightarrow$ auditor verification.
• Figure 5: Epoch-based windowing: events map deterministically to $I_w$ independent of processing order. All events with timestamps in $I_{w_{I}}$ are grouped into the same window.