Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Tool-Assisted Agent on SQL Inspection and Refinement in Real-World Scenarios

Zhongyuan Wang, Richong Zhang, Zhijie Nie, Jaein Kim

TL;DR

A tool-assisted agent framework for SQL inspection and refinement is proposed, equipping the LLM-based agent with two specialized tools: a retriever and a detector, designed to diagnose and correct SQL queries with database mismatches.

Abstract

Recent Text-to-SQL methods leverage large language models (LLMs) by incorporating feedback from the database management system. While these methods effectively address execution errors in SQL queries, they struggle with database mismatches -- errors that do not trigger execution exceptions. Database mismatches include issues such as condition mismatches and stricter constraint mismatches, both of which are more prevalent in real-world scenarios. To address these challenges, we propose a tool-assisted agent framework for SQL inspection and refinement, equipping the LLM-based agent with two specialized tools: a retriever and a detector, designed to diagnose and correct SQL queries with database mismatches. These tools enhance the capability of LLMs to handle real-world queries more effectively. We also introduce Spider-Mismatch, a new dataset specifically constructed to reflect the condition mismatch problems encountered in real-world scenarios. Experimental results demonstrate that our method achieves the highest performance on the averaged results of the Spider and Spider-Realistic datasets in few-shot settings, and it significantly outperforms baseline methods on the more realistic dataset, Spider-Mismatch.

Tool-Assisted Agent on SQL Inspection and Refinement in Real-World Scenarios

TL;DR

A tool-assisted agent framework for SQL inspection and refinement is proposed, equipping the LLM-based agent with two specialized tools: a retriever and a detector, designed to diagnose and correct SQL queries with database mismatches.

Abstract

Recent Text-to-SQL methods leverage large language models (LLMs) by incorporating feedback from the database management system. While these methods effectively address execution errors in SQL queries, they struggle with database mismatches -- errors that do not trigger execution exceptions. Database mismatches include issues such as condition mismatches and stricter constraint mismatches, both of which are more prevalent in real-world scenarios. To address these challenges, we propose a tool-assisted agent framework for SQL inspection and refinement, equipping the LLM-based agent with two specialized tools: a retriever and a detector, designed to diagnose and correct SQL queries with database mismatches. These tools enhance the capability of LLMs to handle real-world queries more effectively. We also introduce Spider-Mismatch, a new dataset specifically constructed to reflect the condition mismatch problems encountered in real-world scenarios. Experimental results demonstrate that our method achieves the highest performance on the averaged results of the Spider and Spider-Realistic datasets in few-shot settings, and it significantly outperforms baseline methods on the more realistic dataset, Spider-Mismatch.
Paper Structure (29 sections, 3 equations, 5 figures, 5 tables)

This paper contains 29 sections, 3 equations, 5 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM for Text-to-SQL
  4. LLM-based Agents
  5. Method
  6. LLM-based Text-to-SQL Task
  7. Framework
  8. Function Calls Design
  9. Inspection Tools
  10. Database Retriever
  11. Error Detector
  12. SQL Generation
  13. Spider-Mismatch Dataset
  14. Dataset Construction
  15. Condition Post-processing Module
  16. ...and 14 more sections

Figures (5)

  • Figure 1: A real-world example of the Text-to-SQL task. Compared to execution errors, diagnosing database mismatches in SQL queries is more challenging.
  • Figure 2: Python functions (left) that we designed based on the SELECT statement, along with an example (right) that includes an SQL query and the corresponding actions.
  • Figure 3: The overall structure of Tool-SQL, consisting of an LLM-based agent for SQL refinement (left) and two tools for error inspection (right). The LLM-based agent performs actions to construct the SQL query and continuously refines these actions based on feedback from the tools. The tools target different problems in SQL queries: (i) the Database Retriever detects the condition mismatches, and (ii) the Error Detector diagnoses execution errors and mismatches related to stricter constraints.
  • Figure 4: Ablation study on Spider-Mismatch equipped with different tools. "w/ Retriever + Detector" corresponds to our full method. "w/ DBMS" denotes obtaining feedback by executing SQL queries on the database management system rather than utilizing our detector.
  • Figure 5: The impact of the maximum number of iterations on the performance of Tool-SQL.