Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration

TL;DR

DySQL-Bench addresses the gap in evaluating Text-to-SQL in dynamic, multi-turn scenarios by introducing a two-stage automatic task-synthesis pipeline and a human-verified 1,072-task dataset spanning 13 domains. The benchmark employs a simulated user–model–database interaction framework and an executable-SQL evaluation via state hashing to capture evolving intents and CRUD-based reasoning. Empirical results show that even strong models like GPT-4o face substantial challenges in dynamic SQL with noticeable hallucinations and instability, and gains from scaling alone diminish beyond about 70B parameters. The work provides a rigorous platform for assessing interactive database intelligence and highlights the need for stability, schema-aware reasoning, and pragmatic interaction strategies in real-world deployments.

Abstract

Recent advances in Text-to-SQL have achieved strong results in static, single-turn tasks, where models generate SQL queries from natural language questions. However, these systems fall short in real-world interactive scenarios, where user intents evolve and queries must be refined over multiple turns. In applications such as finance and business analytics, users iteratively adjust query constraints or dimensions based on intermediate results. To evaluate such dynamic capabilities, we introduce DySQL-Bench, a benchmark assessing model performance under evolving user interactions. Unlike previous manually curated datasets, DySQL-Bench is built through an automated two-stage pipeline of task synthesis and verification. Structured tree representations derived from raw database tables guide LLM-based task generation, followed by interaction-oriented filtering and expert validation. Human evaluation confirms 100% correctness of the synthesized data. We further propose a multi-turn evaluation framework simulating realistic interactions among an LLM-simulated user, the model under test, and an executable database. The model must adapt its reasoning and SQL generation as user intents change. DySQL-Bench covers 13 domains across BIRD and Spider 2 databases, totaling 1,072 tasks. Even GPT-4o attains only 58.34% overall accuracy and 23.81% on the Pass@5 metric, underscoring the benchmark's difficulty. All code and data are released at https://github.com/Aurora-slz/Real-World-SQL-Bench .

Figures (11)

• Figure 1: Overview of interaction types between user, LLM agent, and database. From left to right: (a) direct Text-to-SQL execution, (b) iterative reasoning within a single query (ReAct), and (c) our approach enabling multi-step, multi-turn contextual execution.
• Figure 2: Task Generation Pipeline. Based on 13 databases from Spider 2 and BIRD, we designed a two-stage DynSQL-Bench task synthesis method. In the first stage, an LLM generates initial tasks. In the second stage, a filtering mechanism selects high-quality tasks.
• Figure 3: Database Schema Representation.
• Figure 4: Overview of the DySQL-Bench.
• Figure 5: Schematic diagram of dynamic multi-turn interactions among the three roles of User, Agent, and Database.