Supporting Better Insights of Data Science Pipelines with Fine-grained Provenance

TL;DR

The paper tackles the need for fine-grained provenance in data science preprocessing to explain how each transformation affects data and model outcomes. It introduces a formal model of data, manipulation, and provenance, and defines a template-based, change-driven approach to generate PROV-compliant provenance (provlets) by observing input–output changes. A Python prototype with a Neo4j backend implements the provgen templates for core operator classes (reduction, augmentation, transformation, fusion) and assembles complete provenance documents. Empirical evaluation on real pipelines and synthetic data demonstrates the approach’s ability to answer provenance queries and support pipeline debugging, while also discussing performance and extension opportunities. The work enables targeted data introspection, auditing, and design guidance for data science pipelines, with practical impact on transparency and reliability of ML workflows.

Abstract

Successful data-driven science requires complex data engineering pipelines to clean, transform, and alter data in preparation for machine learning, and robust results can only be achieved when each step in the pipeline can be justified, and its effect on the data explained. In this framework, our aim is to provide data scientists with facilities to gain an in-depth understanding of how each step in the pipeline affects the data, from the raw input to training sets ready to be used for learning. Starting from an extensible set of data preparation operators commonly used within a data science setting, in this work we present a provenance management infrastructure for generating, storing, and querying very granular accounts of data transformations, at the level of individual elements within datasets whenever possible. Then, from the formal definition of a core set of data science preprocessing operators, we derive a provenance semantics embodied by a collection of templates expressed in PROV, a standard model for data provenance. Using those templates as a reference, our provenance generation algorithm generalises to any operator with observable input/output pairs. We provide a prototype implementation of an application-level provenance capture library to produce, in a semi-automatic way, complete provenance documents that account for the entire pipeline. We report on the ability of our implementations to capture provenance in real ML benchmark pipelines and over TCP-DI synthetic data. We finally show how the collected provenance can be used to answer a suite of provenance benchmark queries that underpin some common pipeline inspection questions, as expressed on the Data Science Stack Exchange.

Figures (21)

• Figure 1: The core W3C PROV model.
• Figure 2: A fragment of provenance data for the operation in Example \ref{['ex:ProvModel']}.
• Figure 3: Example of PROV template for Vertical Augmentation and corresponding instances.
• Figure 4: PROV templates used by the prov-gen functions for data augmentation, transformation, and reduction.
• Figure 5: Provlets describing grouping and aggregation from Example in Sec.\ref{['sec:ha']}

Theorems & Definitions (8)

• Example 2.1
• Example 2.2
• Example 2.3
• Example 2.4
• Example 2.5
• Example 2.6
• Example 5.1: one-hot encoding
• Example 5.2: Provenance of one-hot encoding