Beyond SELECT: A Comprehensive Taxonomy-Guided Benchmark for Real-World Text-to-SQL Translation

TL;DR

This work introduces a four-dimension taxonomy for Text-to-SQL data and a taxonomy-guided synthesis pipeline to construct SQL-Synth, a large, diverse benchmark surpassing existing datasets in coverage. The approach reveals gaps in Spider and Bird and demonstrates that LLM-based data generation benefits from taxonomy-guided constraints and enhanced databases. SQL-Synth, containing over 100k high-quality samples, enables robust evaluation and targeted fine-tuning of models, exemplified by Synth-Coder, which achieves state-of-the-art performance with fewer parameters. The study underscores the taxonomy's potential to analyze datasets, benchmark diverse LLMs, and guide scalable training data construction for real-world Text-to-SQL applications.

Abstract

Text-to-SQL datasets are essential for training and evaluating text-to-SQL models, but existing datasets often suffer from limited coverage and fail to capture the diversity of real-world applications. To address this, we propose a novel taxonomy for text-to-SQL classification based on dimensions including core intents, statement types, syntax structures, and key actions. Using this taxonomy, we evaluate widely used public text-to-SQL datasets (e.g., Spider and Bird) and reveal limitations in their coverage and diversity. We then introduce a taxonomy-guided dataset synthesis pipeline, yielding a new dataset named SQL-Synth. This approach combines the taxonomy with Large Language Models (LLMs) to ensure the dataset reflects the breadth and complexity of real-world text-to-SQL applications. Extensive analysis and experimental results validate the effectiveness of our taxonomy, as SQL-Synth exhibits greater diversity and coverage compared to existing benchmarks. Moreover, we uncover that existing LLMs typically fall short in adequately capturing the full range of scenarios, resulting in limited performance on SQL-Synth. However, fine-tuning can substantially improve their performance in these scenarios. The proposed taxonomy has significant potential impact, as it not only enables comprehensive analysis of datasets and the performance of different LLMs, but also guides the construction of training data for LLMs.

Figures (8)

• Figure 1: The proposed taxonomy for text-to-SQL. The taxonomy consists of four main dimensions, including Core intents, Statement types, Syntax structures, and Key actions. Core intents focus on the underlying purpose of the user question. In contrast, Statement types, Syntax structures, and Key actions emphasize the specific implementation details from the perspective of the resulting SQL query.
• Figure 2: Taxonomic distribution of public Spider and Bird datasets.
• Figure 3: The taxonomy-guided dataset synthesis pipeline consists of four main processes: (i) Complexity-aware taxonomy combination, which generates valid taxonomy combinations under various complexity-level; (ii) Database enhancement to generate meaningful databases; (iii) Seed data generation, which produces high-quality seed data that serves as templates for diversity expansion; (iv) Dual-path diversity expansion, where LLMs expand the diversity of seed data leveraging the enhanced databases.
• Figure 4: Evaluations on statement type.
• Figure 5: Quality evaluation of SQL-Synth and Spider judged by GPT-4o. Scores are computed using Equation \ref{['score equation']}.