DataGovBench: Benchmarking LLM Agents for Real-World Data Governance Workflows

TL;DR

DataGovbench introduces a hierarchical, real-world benchmark for NL-to-data-governance automation, addressing the gap in end-to-end evaluation of data-quality pipelines. It leverages a reversed-objective noise synthesis strategy and task-specific evaluation scripts to quantify correctness with CRR, TSR, and ATS, revealing significant gaps in current LLM agents. The authors propose DataGovAgent, a Planner-Executor-Evaluator framework with contract-guided planning, retrieval-augmented generation, and sandboxed debugging, achieving substantial improvements in task success and debugging efficiency on operator- and DAG-level tasks. The work advances practical governance automation by providing a rigorous, reproducible evaluation suite and an end-to-end agent architecture that better aligns runnable code with business objectives, albeit with acknowledged scalability and efficiency trade-offs.

Abstract

Data governance ensures data quality, security, and compliance through policies and standards, a critical foundation for scaling modern AI development. Recently, large language models (LLMs) have emerged as a promising solution for automating data governance by translating user intent into executable transformation code. However, existing benchmarks for automated data science often emphasize snippet-level coding or high-level analytics, failing to capture the unique challenge of data governance: ensuring the correctness and quality of the data itself. To bridge this gap, we introduce DataGovBench, a benchmark featuring 150 diverse tasks grounded in real-world scenarios, built on data from actual cases. DataGovBench employs a novel "reversed-objective" methodology to synthesize realistic noise and utilizes rigorous metrics to assess end-to-end pipeline reliability. Our analysis on DataGovBench reveals that current models struggle with complex, multi-step workflows and lack robust error-correction mechanisms. Consequently, we propose DataGovAgent, a framework utilizing a Planner-Executor-Evaluator architecture that integrates constraint-based planning, retrieval-augmented generation, and sandboxed feedback-driven debugging. Experimental results show that DataGovAgent significantly boosts the Average Task Score (ATS) on complex tasks from 39.7 to 54.9 and reduces debugging iterations by over 77.9 percent compared to general-purpose baselines.

Figures (10)

• Figure 1: Distribution of the 100 operator-level tasks in DataGovbench across six governance categories: Filtering (22), Refinement (18), Imputation (18), Deduplication & Consistency (15), Integration (18), and Classification & Labeling (9). The split balances coverage of common governance operations while slightly emphasizing filtering and transformation tasks.
• Figure 2: Illustration of the semi-automated pipeline designed for building DataGovbench, including real-world source data curation, hierarchical task objective design, noisy data synthesis, task-specific evaluation and DAG-level task construction.
• Figure 3: An overview of the Agentic Assembly Line. The process progresses from intent understanding to contract-guided planning. Unlike traditional layouts, this figure spans roughly 1.5 columns, allowing the caption to sit snugly on the right side, creating a wrapping effect efficiently.
• Figure 4: Performance of Agent Framework Baselines
• Figure 5: BenchDemo visual examples in a compact layout.