Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PTD-SQL: Partitioning and Targeted Drilling with LLMs in Text-to-SQL

Ruilin Luo, Liyuan Wang, Binghuai Lin, Zicheng Lin, Yujiu Yang

TL;DR

It is proposed that employing query group partitioning allows LLMs to focus on learning the thought processes specific to a single problem type, consequently enhancing their reasoning abilities across diverse difficulty levels and problem categories.

Abstract

Large Language Models (LLMs) have emerged as powerful tools for Text-to-SQL tasks, exhibiting remarkable reasoning capabilities. Different from tasks such as math word problems and commonsense reasoning, SQL solutions have a relatively fixed pattern. This facilitates the investigation of whether LLMs can benefit from categorical thinking, mirroring how humans acquire knowledge through inductive reasoning based on comparable examples. In this study, we propose that employing query group partitioning allows LLMs to focus on learning the thought processes specific to a single problem type, consequently enhancing their reasoning abilities across diverse difficulty levels and problem categories. Our experiments reveal that multiple advanced LLMs, when equipped with PTD-SQL, can either surpass or match previous state-of-the-art (SOTA) methods on the Spider and BIRD datasets. Intriguingly, models with varying initial performances have exhibited significant improvements, mainly at the boundary of their capabilities after targeted drilling, suggesting a parallel with human progress. Code is available at https://github.com/lrlbbzl/PTD-SQL.

PTD-SQL: Partitioning and Targeted Drilling with LLMs in Text-to-SQL

TL;DR

It is proposed that employing query group partitioning allows LLMs to focus on learning the thought processes specific to a single problem type, consequently enhancing their reasoning abilities across diverse difficulty levels and problem categories.

Abstract

Large Language Models (LLMs) have emerged as powerful tools for Text-to-SQL tasks, exhibiting remarkable reasoning capabilities. Different from tasks such as math word problems and commonsense reasoning, SQL solutions have a relatively fixed pattern. This facilitates the investigation of whether LLMs can benefit from categorical thinking, mirroring how humans acquire knowledge through inductive reasoning based on comparable examples. In this study, we propose that employing query group partitioning allows LLMs to focus on learning the thought processes specific to a single problem type, consequently enhancing their reasoning abilities across diverse difficulty levels and problem categories. Our experiments reveal that multiple advanced LLMs, when equipped with PTD-SQL, can either surpass or match previous state-of-the-art (SOTA) methods on the Spider and BIRD datasets. Intriguingly, models with varying initial performances have exhibited significant improvements, mainly at the boundary of their capabilities after targeted drilling, suggesting a parallel with human progress. Code is available at https://github.com/lrlbbzl/PTD-SQL.
Paper Structure (45 sections, 5 equations, 7 figures, 23 tables)

This paper contains 45 sections, 5 equations, 7 figures, 23 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Reasoning
  4. LLM-based Text-to-SQL
  5. Pipeline of PTD-SQL
  6. Query Group Partition
  7. Problem Formulation
  8. Fine-tuned LLM Classifier
  9. Finetuned Small Model vs. Few-shot GPT
  10. Targeted Drilling Bank Auto-construction
  11. Few-shot Selection
  12. Semantic matching
  13. Syntactic matching
  14. Mix-of-matching
  15. Experiments
  16. ...and 30 more sections

Figures (7)

  • Figure 1: Demonstration of targeted drilling prompt on multi-set problems.
  • Figure 2: Some samples of proposed partition.
  • Figure 3: Prompt demonstrations for Multi-set and Combination problem.
  • Figure 4: Overflow of PTD-SQL. (a) QGP sub-task. (b) Targeted drilling bank auto-construction. (c) Reasoning step.
  • Figure 5: Under different difficulty levels, the percentage gain (%) in EX metric on Spider (left) and BIRD (right) obtained by the three models using PTD-SQL compared to DIN-SQL.
  • ...and 2 more figures