Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Trustworthy AI in the Agentic Lakehouse: from Concurrency to Governance

Jacopo Tagliabue, Federico Bianchi, Ciro Greco

TL;DR

The paper addresses the challenge of making production AI agents trustworthy on lakehouses by focusing on data and compute isolation rather than relying on traditional MVCC mappings. It introduces Bauplan, an agent-first lakehouse design with copy-on-write branching, FaaS compute, and a unified run API to enable transactional pipelines across multi-language workloads. It argues that this approach provides principled governance through API-level access control and declarative I/O, illustrated with a self-healing pipeline example. The work offers a practical reference implementation and lays the groundwork for scalable, trustworthy agent workflows in data platforms.

Abstract

Even as AI capabilities improve, most enterprises do not consider agents trustworthy enough to work on production data. In this paper, we argue that the path to trustworthy agentic workflows begins with solving the infrastructure problem first: traditional lakehouses are not suited for agent access patterns, but if we design one around transactions, governance follows. In particular, we draw an operational analogy to MVCC in databases and show why a direct transplant fails in a decoupled, multi-language setting. We then propose an agent-first design, Bauplan, that reimplements data and compute isolation in the lakehouse. We conclude by sharing a reference implementation of a self-healing pipeline in Bauplan, which seamlessly couples agent reasoning with all the desired guarantees for correctness and trust.

Trustworthy AI in the Agentic Lakehouse: from Concurrency to Governance

TL;DR

The paper addresses the challenge of making production AI agents trustworthy on lakehouses by focusing on data and compute isolation rather than relying on traditional MVCC mappings. It introduces Bauplan, an agent-first lakehouse design with copy-on-write branching, FaaS compute, and a unified run API to enable transactional pipelines across multi-language workloads. It argues that this approach provides principled governance through API-level access control and declarative I/O, illustrated with a self-healing pipeline example. The work offers a practical reference implementation and lays the groundwork for scalable, trustworthy agent workflows in data platforms.

Abstract

Even as AI capabilities improve, most enterprises do not consider agents trustworthy enough to work on production data. In this paper, we argue that the path to trustworthy agentic workflows begins with solving the infrastructure problem first: traditional lakehouses are not suited for agent access patterns, but if we design one around transactions, governance follows. In particular, we draw an operational analogy to MVCC in databases and show why a direct transplant fails in a decoupled, multi-language setting. We then propose an agent-first design, Bauplan, that reimplements data and compute isolation in the lakehouse. We conclude by sharing a reference implementation of a self-healing pipeline in Bauplan, which seamlessly couples agent reasoning with all the desired guarantees for correctness and trust.

Paper Structure

This paper contains 16 sections, 3 figures.

Table of Contents

  1. Introduction
  2. The MVCC mental model
  3. Data Isolation
  4. Compute Isolation
  5. Programming Abstractions
  6. From MVCC to the lakehouse
  7. Data Isolation
  8. Compute Isolation
  9. Programming Abstractions
  10. The Agentic Lakehouse
  11. Data Isolation
  12. Compute Isolation
  13. Programming Abstractions
  14. From concurrency control to governance
  15. A worked-out example: self-healing pipelines
  16. ...and 1 more sections

Figures (3)

  • Figure 1: The MVCC mental model: $U_1$ starts his transaction, which at the end returns the value of $B$ -- effectively, his code works as if$U_2$'s transaction had never happened.
  • Figure 2: Transactional pipelines.Top: without coupling temporary branches with pipeline runs, run_2 will leave in main a new version of $A$ but an old version of $B$. Bottom: Bauplanrun API will guarantee atomic write of $A'$ and $B'$ on success -- run_1 --, and isolation in case of failure -- run_2.
  • Figure 3: Self-healing pipelines: a ReACT loop is triggered on the agentic lakehouse -- at the end, a verifier acts as a first sanity check on the agent work, leaving to a human the final confirmation thanks to the branch-then-merge flow.