Reproducible data science over data lakes: replayable data pipelines with Bauplan and Nessie

TL;DR

The paper tackles reproducibility of data workloads over data lakes by decoupling compute from data management using Bauplan and the Nessie data catalog. It presents a multi-language, declarative pipeline model executed in a cloud FaaS runtime, with time-travel and branching semantics enabled by a Git-like data catalog. Key contributions include abstractions across in-memory to on-disk representations (Arrow, Parquet, Iceberg), a simple CLI for cloud execution, and Nessie’s branching/transactions for reproducible runs. The approach aims to reduce debugging complexity and provide reproducible pipelines with minimal user effort, enhancing practicality for Lakehouse-style architectures.

Abstract

As the Lakehouse architecture becomes more widespread, ensuring the reproducibility of data workloads over data lakes emerges as a crucial concern for data engineers. However, achieving reproducibility remains challenging. The size of data pipelines contributes to slow testing and iterations, while the intertwining of business logic and data management complicates debugging and increases error susceptibility. In this paper, we highlight recent advancements made at Bauplan in addressing this challenge. We introduce a system designed to decouple compute from data management, by leveraging a cloud runtime alongside Nessie, an open-source catalog with Git semantics. Demonstrating the system's capabilities, we showcase its ability to offer time-travel and branching semantics on top of object storage, and offer full pipeline reproducibility with a few CLI commands.

Figures (4)

• Figure 1: Interleaving of compute (EC2) and storage (S3) in a data pipeline, modelled as a Directed Acyclic Graph (DAG): the responsibility of the data scientist is to write functions that transform the original data artifact (the source data) in intermediate and then final dataframes for downstream consumption (e.g. run a ML model).
• Figure 2: Hierarchy of data representation: while data scientists interact only with in-process dataframes, the system persists the information preserving the overall semantics through different (reversible) layers of abstraction -- physical files, collection of files into tables, collection of tables.
• Figure 3: Data flow for a pipeline run: 1) user issues a query to the middleware, which 2) sends a plan to the runtime; 3) the runtime asks Nessie for the parquet files backing the plan and 4) retrieves them from S3. Finally, the pipeline is run and 5) results are sent back to the client.
• Figure 4: Debugging on Bauplan: when Richard reproduces Monday's run, the system 1) travels back in time at Monday's source data and pipeline code, 2) creates a debug branch for his experiments, and 3) materializes the target artifacts inside the branch.