Does Provenance Interact?

TL;DR

The paper tackles the scalability challenge of data provenance in large-scale, time-evolving dataflows by proposing Temporal Interaction Networks (TINs), a graph-based model that captures time, flow, and state with explicit quantities. It formalizes a unified provenance representation G = (V, E, R) with interactions r = (r_s, r_d, r_t, r_q) and per-vertex buffers, enabling precise temporal and volumetric provenance tracking across streaming, transportation, and financial networks. A discrete-vs-liquid data classification, five temporal provenance query types (Backward, Forward, Temporal Lineage, Flow Lineage, Versioning), and a state-based indexing approach are introduced to support efficient, history-agnostic queries and substantial storage compression. The work outlines a path toward practical, scalable temporal provenance for large dataflows by identifying optimization, compression, and distributed indexing research directions with concrete success metrics. Overall, this approach has the potential to transform temporal provenance from a debugging aid into a robust data-management primitive capable of handling real-time, large-scale provenance queries.

Abstract

Data provenance (the process of determining the origin and derivation of data outputs) has applications across multiple domains including explaining database query results and auditing scientific workflows. Despite decades of research, provenance tracing remains challenging due to computational costs and storage overhead. In streaming systems such as Apache Flink, provenance graphs can grow super-linearly with data volume, posing significant scalability challenges. Temporal provenance is a promising direction, attaching timestamps to provenance information, enabling time-focused queries without maintaining complete historical records. However, existing temporal provenance methods primarily focus on system-level debugging, leaving a gap in data management applications. This paper proposes an agenda that uses Temporal Interaction Networks (TINs) to represent temporal provenance efficiently. We demonstrate TINs' applicability across streaming systems, transportation networks, and financial networks. We classify data into discrete and liquid types, define five temporal provenance query types, and propose a state-based indexing approach. Our vision outlines research directions toward making temporal provenance a practical tool for large-scale dataflows.

Figures (1)

• Figure 1: TIN-based provenance framework.