NL2SQL-BUGs: A Benchmark for Detecting Semantic Errors in NL2SQL Translation

TL;DR

The paper tackles the challenge of semantic errors in NL2SQL translations by proposing NL2SQL-BUGs, the first benchmark dedicated to detecting and categorizing semantic mistakes. It introduces a two-level taxonomy with 9 main categories and 31 subcategories, and provides 2,018 expert-annotated NL-DB-SQL triples (1,019 correct, 999 incorrect). Empirical results show current LLMs achieve about 75–77% accuracy on semantic error detection, with some models detecting previously overlooked errors in Spider and BIRD benchmarks. The authors detail a three-step construction pipeline for NL2SQL-BUGs, analyze model strengths and weaknesses across error types, and advocate for deeper semantic-error detection to enable robust, deployable NL2SQL systems through cross-domain collaboration.

Abstract

Natural Language to SQL (i.e., NL2SQL) translation is crucial for democratizing database access, but even state-of-the-art models frequently generate semantically incorrect SQL queries, hindering the widespread adoption of these techniques by database vendors. While existing NL2SQL benchmarks primarily focus on correct query translation, we argue that a benchmark dedicated to identifying common errors in NL2SQL translations is equally important, as accurately detecting these errors is a prerequisite for any subsequent correction-whether performed by humans or models. To address this gap, we propose NL2SQL-BUGs, the first benchmark dedicated to detecting and categorizing semantic errors in NL2SQL translation. NL2SQL-BUGs adopts a two-level taxonomy to systematically classify semantic errors, covering 9 main categories and 31 subcategories. The benchmark consists of 2,018 expert-annotated instances, each containing a natural language query, database schema, and SQL query, with detailed error annotations for semantically incorrect queries. Through comprehensive experiments, we demonstrate that current large language models exhibit significant limitations in semantic error detection, achieving an average detection accuracy of 75.16%. Specifically, our method successfully detected 106 errors (accounting for 6.91%) in BIRD, a widely-used NL2SQL dataset, which were previously undetected annotation errors. This highlights the importance of semantic error detection in NL2SQL systems. The benchmark is publicly available at https://nl2sql-bugs.github.io/.

Figures (6)

• Figure 1: Error Types in Generated NL2SQL Queries.
• Figure 2: Semantic Errors Detection for NL2SQL.
• Figure 3: A Taxonomy of NL2SQL Translation Semantic Errors.
• Figure 4: Data Structure and Proportion of Error Types in NL2SQL-BUGs Benchmark.
• Figure 5: The Construction Pipeline of NL2SQL-BUGs.