Evaluating and Enhancing LLMs for Multi-turn Text-to-SQL with Multiple Question Types

TL;DR

This work proposes MMSQL, a comprehensive test suite designed to evaluate LLMs’ question classification and SQL generation capabilities by simulating real-world scenarios with diverse question types and multi-turn Q&A interactions, and introduces an LLM-based multi-agent framework that employs specialized agents to identify question types and determine appropriate answering strategies.

Abstract

Recent advancements in large language models (LLMs) have significantly advanced text-to-SQL systems. However, most LLM-based methods often narrowly focus on SQL generation, neglecting the complexities of real-world conversational queries. This oversight can lead to unreliable responses, particularly for ambiguous questions that cannot be directly addressed with SQL. To bridge this gap, we propose MMSQL, a comprehensive test suite designed to evaluate the question classification and SQL generation capabilities of LLMs by simulating real-world scenarios with diverse question types and multi-turn Q&A interactions. Using MMSQL, we assessed the performance of popular LLMs, including both open-source and closed-source models, and identified key factors impacting their performance in such scenarios. Moreover, we introduce an LLM-based multi-agent framework that employs specialized agents to identify question types and determine appropriate answering strategies. Our experiments demonstrate that this approach significantly enhances the model's ability to navigate the complexities of conversational dynamics, effectively handling the diverse and complex nature of user queries. Our dataset and code are publicly available at https://mcxiaoxiao.github.io/MMSQL.

Figures (6)

• Figure 1: A four-turn dialogue example that includes four different types of Q&A pairs.
• Figure 2: Distribution of question types in the curated MMSQL dataset, highlighting the proportions of Answerable, Ambiguous, Improper, and Unanswerable questions in both the training set and the test set.
• Figure 3: Performance analysis of model outputs on different Q&A scenarios in the MMSQL test set
• Figure 4: The overview of our multi-agent framework comprises four components: (i) the Schema Selector, which narrows down the database schema to focus on relevant tables, reducing noise from irrelevant data; (ii) the Question Detector, which determines its type and reformulates it if the question is deemed potentially ambiguous, potentially generating multiple possible rewrites; (iii) the Question Decomposer, which breaks down complex questions into simpler, manageable sub-questions and applies chain-of-thought reasoning to solve them progressively; and (iv) the SQL Refiner, which utilizes an external tool for SQL execution, gathers feedback, then refines faulty SQL queries.
• Figure 5: Overview of QDA-SQL processes.