Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DB-GPT-Hub: Towards Open Benchmarking Text-to-SQL Empowered by Large Language Models

Fan Zhou, Siqiao Xue, Danrui Qi, Wenhui Shi, Wang Zhao, Ganglin Wei, Hongyang Zhang, Caigai Jiang, Gangwei Jiang, Zhixuan Chu, Faqiang Chen

TL;DR

This work introduces DB-GPT-Hub, an open benchmark framework for LLM-empowered text-to-SQL that prioritizes tuning large open LLMs. By standardizing data processing, evaluation, and a modular codebase, it enables end-to-end, reproducible comparisons between prompting and fine-tuning approaches, with emphasis on parameter-efficient fine-tuning via LoRA and QLoRA. Empirical results show tuned CodeLlama variants often outperform prompting baselines, especially on cross-domain datasets like Spider, while revealing tradeoffs in compute time and memory between LoRA and QLoRA. The study provides actionable insights into when tuning yields the most benefit (e.g., easier SQL tasks) and highlights the importance of training corpus diversity. Overall, DB-GPT-Hub offers a scalable, extensible platform to accelerate research and application of open benchmarks for text-to-SQL with large language models.

Abstract

Large language models (LLMs) becomes the dominant paradigm for the challenging task of text-to-SQL. LLM-empowered text-to-SQL methods are typically categorized into prompting-based and tuning approaches. Compared to prompting-based methods, benchmarking fine-tuned LLMs for text-to-SQL is important yet under-explored, partially attributed to the prohibitively high computational cost. In this paper, we present DB-GPT-Hub, an open benchmark suite for LLM-empowered text-to-SQL, which primarily focuses on tuning LLMs at large scales. The proposed benchmark consists of: 1. a standardized and comprehensive evaluation of text-to-SQL tasks by fine-tuning medium to large-sized open LLMs; 2. a modularized and easy-to-extend codebase with mainstream LLMs and experimental scenarios supported, which prioritizes fine-tuning methods but can be easily extended to prompt-based setting. Our work investigates the potential gains and the performance boundaries of tuning approaches, compared to prompting approaches and explores optimal solutions tailored to specific scenarios. We hope DB-GPT-Hub, along with these findings, enables further research and broad applications that would otherwise be difficult owing to the absence of a dedicated open benchmark. The project code has been released at https://github.com/eosphoros-ai/DB-GPT-Hub.

DB-GPT-Hub: Towards Open Benchmarking Text-to-SQL Empowered by Large Language Models

TL;DR

This work introduces DB-GPT-Hub, an open benchmark framework for LLM-empowered text-to-SQL that prioritizes tuning large open LLMs. By standardizing data processing, evaluation, and a modular codebase, it enables end-to-end, reproducible comparisons between prompting and fine-tuning approaches, with emphasis on parameter-efficient fine-tuning via LoRA and QLoRA. Empirical results show tuned CodeLlama variants often outperform prompting baselines, especially on cross-domain datasets like Spider, while revealing tradeoffs in compute time and memory between LoRA and QLoRA. The study provides actionable insights into when tuning yields the most benefit (e.g., easier SQL tasks) and highlights the importance of training corpus diversity. Overall, DB-GPT-Hub offers a scalable, extensible platform to accelerate research and application of open benchmarks for text-to-SQL with large language models.

Abstract

Large language models (LLMs) becomes the dominant paradigm for the challenging task of text-to-SQL. LLM-empowered text-to-SQL methods are typically categorized into prompting-based and tuning approaches. Compared to prompting-based methods, benchmarking fine-tuned LLMs for text-to-SQL is important yet under-explored, partially attributed to the prohibitively high computational cost. In this paper, we present DB-GPT-Hub, an open benchmark suite for LLM-empowered text-to-SQL, which primarily focuses on tuning LLMs at large scales. The proposed benchmark consists of: 1. a standardized and comprehensive evaluation of text-to-SQL tasks by fine-tuning medium to large-sized open LLMs; 2. a modularized and easy-to-extend codebase with mainstream LLMs and experimental scenarios supported, which prioritizes fine-tuning methods but can be easily extended to prompt-based setting. Our work investigates the potential gains and the performance boundaries of tuning approaches, compared to prompting approaches and explores optimal solutions tailored to specific scenarios. We hope DB-GPT-Hub, along with these findings, enables further research and broad applications that would otherwise be difficult owing to the absence of a dedicated open benchmark. The project code has been released at https://github.com/eosphoros-ai/DB-GPT-Hub.
Paper Structure (49 sections, 2 equations, 4 figures, 9 tables)

This paper contains 49 sections, 2 equations, 4 figures, 9 tables.

Table of Contents

  1. Introduction
  2. Background and Problem Formulation
  3. A Generalized Setup
  4. Prompt-based and Fine-tuning Settings
  5. Zero-shot / Few-shot Prompting.
  6. Fine-tuning.
  7. Parameter-Efficient Fine-tuning.
  8. Benchmark Design and Resources
  9. Setup
  10. Datasets.
  11. Query-response Construction.
  12. Metrics.
  13. Base LLMs.
  14. Tuning Methods.
  15. Benchmark Pipeline.
  16. ...and 34 more sections

Figures (4)

  • Figure 1: An open benchmarking pipeline using DB-GPT-Hub.
  • Figure 2: The improvement on tuning with LoRA strategy across subgroups of different complexities.
  • Figure 3: Few-shot evaluations on Spider: EX improvement on few-shot scenarios over zero-shot. EX(k-shot) represents the EX of the target (untuned/tuned) model under k-shot scenario minus EX of the base model in zero-shot scenario, i.e., in (a), Improvement on EX(Qwen-LoRA, 3-shot) = EX(Qwen-LoRA, 3-shot) - EX(Qwen, 0-shot).
  • Figure 4: Few-shot evaluations on Spider: the EX performance of Llama2 / Qwen and their tuned counterparts with varying model size.