SQLFixAgent: Towards Semantic-Accurate Text-to-SQL Parsing via Consistency-Enhanced Multi-Agent Collaboration

TL;DR

This work addresses semantic inaccuracies in Text-to-SQL produced by fine-tuned LLMs by introducing SQLFixAgent, a consistency-enhanced multi-agent system that coordinates SQLReviewer, QueryCrafter, and SQLRefiner to detect and repair erroneous SQL. The framework leverages Rubber Duck Debugging for semantic checks, generates diverse candidate SQL via input perturbations, and recalls similar repairs through failure memory to select the best semantically aligned query. Evaluations on Bird, Spider, and robustness variants show consistent improvements in execution accuracy and competitive token efficiency, even with smaller backbone models. The approach demonstrates that combining a foundation LLM with task-specialized agents yields more reliable semantic results and practical performance for real-world Text-to-SQL tasks, with potential extensions to knowledge graphs.

Abstract

While fine-tuned large language models (LLMs) excel in generating grammatically valid SQL in Text-to-SQL parsing, they often struggle to ensure semantic accuracy in queries, leading to user confusion and diminished system usability. To tackle this challenge, we introduce SQLFixAgent, a new consistency-enhanced multi-agent collaborative framework designed for detecting and repairing erroneous SQL. Our framework comprises a core agent, SQLRefiner, alongside two auxiliary agents: SQLReviewer and QueryCrafter. The SQLReviewer agent employs the rubber duck debugging method to identify potential semantic mismatches between SQL and user query. If the error is detected, the QueryCrafter agent generates multiple SQL as candidate repairs using a fine-tuned SQLTool. Subsequently, leveraging similar repair retrieval and failure memory reflection, the SQLRefiner agent selects the most fitting SQL statement from the candidates as the final repair. We evaluated our proposed framework on five Text-to-SQL benchmarks. The experimental results show that our method consistently enhances the performance of the baseline model, specifically achieving an execution accuracy improvement of over 3% on the Bird benchmark. Our framework also has a higher token efficiency compared to other advanced methods, making it more competitive.

Figures (6)

• Figure 1: A complex example of Text-to-SQL from Bird. We demonstrate two common types of errors happened in LLM-based parsing: SQL with syntax errors, where missing table names are easily detected, while SQL with semantic errors is often executed, leading to user confusion.
• Figure 2: The overview of SQLFixAgent framework, which comprises three agents: (i) the SQLReviewer, which detects syntax and semantic errors in SQL generated by SQLTool based on user query and schema sequence, (ii) the QueryCrafter, which generates multiple variants of user query, then utilizes SQLTool to generate multiple SQL as candidate repairs, and (iii) the SQLRefiner, which records runtime errors from SQLTool, and selects the optimal SQL as final repair from candidates.
• Figure 3: The SQLReviewer Agent Illustration.
• Figure 4: The SQLRefiner Agent Illustration.
• Figure 5: The fine-grained categorical evaluation of backbone models on BIRD’s dev set.