Beyond Text-to-SQL: Can LLMs Really Debug Enterprise ETL SQL?

TL;DR

The paper presents Squirrel Benchmark, the first enterprise-scale SQL debugging benchmark built via an automated reverse-engineering workflow to inject realistic syntax and semantic bugs into real-world ETL SQL scripts. It introduces an execution-free evaluation suite using Exact Match, Graph Match on canonical ASTs, and a repair-progress metric, enabling scalable, private-data-safe assessment. With 469 syntax and 516 semantic tasks, the benchmark captures deep structural complexity and diverse domain contexts, and its validation shows substantial gaps for current LLMs, highlighting the need for improved reasoning and debugging capabilities. The work also demonstrates how SFT and agent-based approaches can improve performance, offering practical guidance for deploying LLMs in enterprise SQL debugging tasks and establishing a strong benchmark-correlated proxy for real-world outcomes.

Abstract

SQL is central to enterprise data engineering, yet generating fully correct SQL code in a single attempt remains difficult, even for experienced developers and advanced text-to-SQL LLMs, often requiring multiple debugging iterations. We introduce OurBench, the first benchmark for enterprise-level SQL reasoning and debugging. Our benchmark is built on two key innovations: (1) an automated construction workflow that uses reverse engineering to systematically inject realistic bugs into large-scale SQL code, enabling scalable and diverse benchmark generation; and (2) an execution-free evaluation framework tailored to enterprise settings, providing fast, accurate, and resource-efficient assessment. OurBench comprises 469 OurBenchSyn queries featuring syntax errors with explicit error messages, and 516 OurBenchSem queries targeting semantic errors in which the code fails to meet user intent. The queries are highly complex, averaging over 140 lines and featuring deep and wide abstract syntax trees. Evaluation of nearly 30 LLMs reveals a substantial performance gap: the best-performing model, Claude-4-Sonnet, achieves only 36.46 percent accuracy on OurBenchSyn and 32.17 percent on OurBenchSem, while most models score below 20 percent. We further explore four solution strategies, identify key challenges, and outline promising directions for enterprise SQL debugging with LLMs.

Figures (15)

• Figure 1: Squirrel Benchmark evaluates LLMs on real-world enterprise-level SQL debugging workflows. It involves multi-step reasoning and actions, including understanding requirements and schemas, diagnosing error messages, and iteratively refining scripts through cycles of reasoning and debugging.
• Figure 2: Overview of the Squirrel Benchmark construction and evaluation pipeline. Benchmark construction consists of 4 main stages: (1) Enterprise-level SQL Script Generation, (2) SQL bug taxonomy Design, (3) Issue SQL Construction via reverse engineering, and (4) Validation and Analysis. This pipeline ensures diversity, realism, and rigorous evaluation of the SQL Debugging task.
• Figure 4: SFT baseline performance on Squirrel-Syntax. The horizontal axis represents the average inference speed, and the vertical axis shows the GM score.
• Figure 5: Agent performance on Squirrel-Syntax. '$*$' denotes agent-based methods, while others are single-model baselines.
• Figure 6: Illustration of Graph Match Score. Although the left and right SQL snippets differ syntactically, their optimized abstract syntax trees are structurally identical. Graph matching evaluates semantic equivalence through tree isomorphism.