Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SQLCritic: Correcting Text-to-SQL Generation via Clause-wise Critic

Jikai Chen, Leilei Gan, Ziyu Zhao, Zechuan Wang, Dong Wang, Chenyi Zhuang

TL;DR

A clause-wise critique generation task along with a benchmark, SQLCriticBench, which performs fine-grained error localization including both syntax and semantic errors at the clause level and an automatically training dataset curation pipeline which annotate clause-wise critique at scale in a cost-effective way.

Abstract

Existing refinement methods in LLM-based Text-to-SQL systems exhibit limited effectiveness. They often introduce new errors during the self-correction process and fail to detect and correct semantic inaccuracies. To address these gaps, we first introduce a clause-wise critique generation task along with a benchmark, SQLCriticBench, which performs fine-grained error localization including both syntax and semantic errors at the clause level. Furthermore, we introduce a variant of DPO for training our SQLCritic model, where the $β$ coefficient is adaptively changed according to the clause-level inconsistencies between the preferred and dispreferred critiques. We also propose an automatically training dataset curation pipeline which annotate clause-wise critique at scale in a cost-effective way. Experiments demonstrate that the SQLCritic model significantly improves SQL accuracy on the BIRD and Spider datasets, and the results on SQLCriticBench further reveals its superior critique capabilities compared to existing models.

SQLCritic: Correcting Text-to-SQL Generation via Clause-wise Critic

TL;DR

A clause-wise critique generation task along with a benchmark, SQLCriticBench, which performs fine-grained error localization including both syntax and semantic errors at the clause level and an automatically training dataset curation pipeline which annotate clause-wise critique at scale in a cost-effective way.

Abstract

Existing refinement methods in LLM-based Text-to-SQL systems exhibit limited effectiveness. They often introduce new errors during the self-correction process and fail to detect and correct semantic inaccuracies. To address these gaps, we first introduce a clause-wise critique generation task along with a benchmark, SQLCriticBench, which performs fine-grained error localization including both syntax and semantic errors at the clause level. Furthermore, we introduce a variant of DPO for training our SQLCritic model, where the coefficient is adaptively changed according to the clause-level inconsistencies between the preferred and dispreferred critiques. We also propose an automatically training dataset curation pipeline which annotate clause-wise critique at scale in a cost-effective way. Experiments demonstrate that the SQLCritic model significantly improves SQL accuracy on the BIRD and Spider datasets, and the results on SQLCriticBench further reveals its superior critique capabilities compared to existing models.

Paper Structure

This paper contains 29 sections, 5 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text-to-SQL Generation.
  4. LLMs Critique.
  5. Clause-wise Text-to-SQL Critique
  6. Task Definition
  7. Training Dataset Curation
  8. Clause-wise Critique Inconsistency Aware Direct Preference Optimization
  9. DPO tailored for clause-wise critique.
  10. Experiment
  11. Experimental Setup
  12. Critiquing and Correcting Results on BIRD and Spider
  13. Clause-wise Critique Evaluation on SQLCritcBench
  14. Ablation Studies
  15. Comparing SFT, DPO and Critic DPO.
  16. ...and 14 more sections

Figures (4)

  • Figure 1: Clause-wise critique in the Text-to-SQL critique and correction task.
  • Figure 2: Overview of SQLCriticBench framework and SQLCritic training process (a1-a2: SQLCriticBench construction process, b: SQLCritic training process).
  • Figure 3: Illustration of filtering low-quality generated SQL query.
  • Figure 4: Error analysis distribution. Different colors represent the distribution of different clauses.