Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing LLM Fine-tuning for Text-to-SQLs by SQL Quality Measurement

Shouvon Sarker, Xishuang Dong, Xiangfang Li, Lijun Qian

TL;DR

A novel approach that only needs SQL Quality Measurement to enhance LLMs-based Text-to-SQLs performance is proposed, which establishes a SQL quality evaluation mechanism to assess the generated SQL queries against predefined criteria and actual database responses.

Abstract

Text-to-SQLs enables non-expert users to effortlessly retrieve desired information from relational databases using natural language queries. While recent advancements, particularly with Large Language Models (LLMs) like GPT and T5, have shown impressive performance on large-scale benchmarks such as BIRD, current state-of-the-art (SOTA) LLM-based Text-to-SQLs models often require significant efforts to develop auxiliary tools like SQL classifiers to achieve high performance. This paper proposed a novel approach that only needs SQL Quality Measurement to enhance LLMs-based Text-to-SQLs performance. It establishes a SQL quality evaluation mechanism to assess the generated SQL queries against predefined criteria and actual database responses. This feedback loop enables continuous learning and refinement of model outputs based on both syntactic correctness and semantic accuracy. The proposed method undergoes comprehensive validation on the BIRD benchmark, assessing Execution Accuracy (EX) and Valid Efficiency Score (VES) across various Text-to-SQLs difficulty levels. Experimental results reveal competitive performance in both EX and VES compared to SOTA models like GPT4 and T5.

Enhancing LLM Fine-tuning for Text-to-SQLs by SQL Quality Measurement

TL;DR

A novel approach that only needs SQL Quality Measurement to enhance LLMs-based Text-to-SQLs performance is proposed, which establishes a SQL quality evaluation mechanism to assess the generated SQL queries against predefined criteria and actual database responses.

Abstract

Text-to-SQLs enables non-expert users to effortlessly retrieve desired information from relational databases using natural language queries. While recent advancements, particularly with Large Language Models (LLMs) like GPT and T5, have shown impressive performance on large-scale benchmarks such as BIRD, current state-of-the-art (SOTA) LLM-based Text-to-SQLs models often require significant efforts to develop auxiliary tools like SQL classifiers to achieve high performance. This paper proposed a novel approach that only needs SQL Quality Measurement to enhance LLMs-based Text-to-SQLs performance. It establishes a SQL quality evaluation mechanism to assess the generated SQL queries against predefined criteria and actual database responses. This feedback loop enables continuous learning and refinement of model outputs based on both syntactic correctness and semantic accuracy. The proposed method undergoes comprehensive validation on the BIRD benchmark, assessing Execution Accuracy (EX) and Valid Efficiency Score (VES) across various Text-to-SQLs difficulty levels. Experimental results reveal competitive performance in both EX and VES compared to SOTA models like GPT4 and T5.
Paper Structure (14 sections, 5 equations, 5 figures, 3 tables)

This paper contains 14 sections, 5 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Task Formulation
  3. Methodology
  4. Step-by-step Prompts
  5. SQL Quality Measurement
  6. Normalized Comparison
  7. Semantic Comparison
  8. Experiments
  9. Dataset
  10. Experiment Setup
  11. Evaluation Metrics
  12. Result and Discussion
  13. Related Work
  14. Conclusion and Future Work

Figures (5)

  • Figure 1: Flow of Enhancing LLM Fine-tuning for Text-to-SQLs by SQL Quality Measurement.
  • Figure 2: Human-designed step-by-step (HDSP) prompts for Text-to-SQLs via LLMs.
  • Figure 3: Flow of the proposed feedback mechanism.
  • Figure 4: An example of data sample from BIRD benchmark.
  • Figure 5: A bar chart provides a clear visualization of the performance of different LLMs configurations.