Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Safe, Untrusted, "Proof-Carrying" AI Agents: toward the agentic lakehouse

Jacopo Tagliabue, Ciro Greco

TL;DR

The work tackles how to enable safe, trustworthy agent-driven automation for data lakehouses by proposing API-first, programmable abstractions that expose the entire data lifecycle. It argues that declarative DAGs, Git-for-Data-like branching, and code-as-interface support reproducibility, observability, and safety, even in the presence of untrusted agents. A proof-of-concept demonstrates self-repair of production pipelines using Bauplan, MCP, and a verifier, showing that untrusted AI agents can operate without compromising production. The study outlines a path toward a fully agentic lakehouse, while identifying future challenges such as scalability and parallelism in OLAP contexts.

Abstract

Data lakehouses run sensitive workloads, where AI-driven automation raises concerns about trust, correctness, and governance. We argue that API-first, programmable lakehouses provide the right abstractions for safe-by-design, agentic workflows. Using Bauplan as a case study, we show how data branching and declarative environments extend naturally to agents, enabling reproducibility and observability while reducing the attack surface. We present a proof-of-concept in which agents repair data pipelines using correctness checks inspired by proof-carrying code. Our prototype demonstrates that untrusted AI agents can operate safely on production data and outlines a path toward a fully agentic lakehouse.

Safe, Untrusted, "Proof-Carrying" AI Agents: toward the agentic lakehouse

TL;DR

The work tackles how to enable safe, trustworthy agent-driven automation for data lakehouses by proposing API-first, programmable abstractions that expose the entire data lifecycle. It argues that declarative DAGs, Git-for-Data-like branching, and code-as-interface support reproducibility, observability, and safety, even in the presence of untrusted agents. A proof-of-concept demonstrates self-repair of production pipelines using Bauplan, MCP, and a verifier, showing that untrusted AI agents can operate without compromising production. The study outlines a path toward a fully agentic lakehouse, while identifying future challenges such as scalability and parallelism in OLAP contexts.

Abstract

Data lakehouses run sensitive workloads, where AI-driven automation raises concerns about trust, correctness, and governance. We argue that API-first, programmable lakehouses provide the right abstractions for safe-by-design, agentic workflows. Using Bauplan as a case study, we show how data branching and declarative environments extend naturally to agents, enabling reproducibility and observability while reducing the attack surface. We present a proof-of-concept in which agents repair data pipelines using correctness checks inspired by proof-carrying code. Our prototype demonstrates that untrusted AI agents can operate safely on production data and outlines a path toward a fully agentic lakehouse.

Paper Structure

This paper contains 10 sections, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. A programmable lakehouse
  3. Pipeline definition
  4. Pipeline execution
  5. Code as the universal interface
  6. The Safety Checklist
  7. A Proof of Concept
  8. The self-repairing setup
  9. Running an experiment
  10. Conclusion and future work

Figures (3)

  • Figure 1: Transactional pipelines: a successful execution of P, followed by one that failed after A materialization. Runs happen on data branches: input data is from sources in main, but writes are sandboxed so that materialized tables hit mainatomically only on success (no half-written pipeline).
  • Figure 2: Sample pipeline: upstream from the pipeline, two source tables containing taxi trips and location data, downstream, multiple consumers. The pipeline itself is a two-node DAG, with compute steps in gray -- 1 and 2 --, and tables in green -- A and B.
  • Figure 3: Safe, untrusted lakehouse agents: the agent leverages LLMs and tools to repair a data pipeline. When an answer is produced, a deterministic check in the "outer loop" verifies that it is safe to merge to production.