Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

XiYan-SQL: A Novel Multi-Generator Framework For Text-to-SQL

Yifu Liu, Yin Zhu, Yingqi Gao, Zhiling Luo, Xiaoxia Li, Xiaorong Shi, Yuntao Hong, Jinyang Gao, Yu Li, Bolin Ding, Jingren Zhou

Abstract

To leverage the advantages of LLM in addressing challenges in the Text-to-SQL task, we present XiYan-SQL, an innovative framework effectively generating and utilizing multiple SQL candidates. It consists of three components: 1) a Schema Filter module filtering and obtaining multiple relevant schemas; 2) a multi-generator ensemble approach generating multiple highquality and diverse SQL queries; 3) a selection model with a candidate reorganization strategy implemented to obtain the optimal SQL query. Specifically, for the multi-generator ensemble, we employ a multi-task fine-tuning strategy to enhance the capabilities of SQL generation models for the intrinsic alignment between SQL and text, and construct multiple generation models with distinct generation styles by fine-tuning across different SQL formats. The experimental results and comprehensive analysis demonstrate the effectiveness and robustness of our framework. Overall, XiYan-SQL achieves a new SOTA performance of 75.63% on the notable BIRD benchmark, surpassing all previous methods. It also attains SOTA performance on the Spider test set with an accuracy of 89.65%.

XiYan-SQL: A Novel Multi-Generator Framework For Text-to-SQL

Abstract

To leverage the advantages of LLM in addressing challenges in the Text-to-SQL task, we present XiYan-SQL, an innovative framework effectively generating and utilizing multiple SQL candidates. It consists of three components: 1) a Schema Filter module filtering and obtaining multiple relevant schemas; 2) a multi-generator ensemble approach generating multiple highquality and diverse SQL queries; 3) a selection model with a candidate reorganization strategy implemented to obtain the optimal SQL query. Specifically, for the multi-generator ensemble, we employ a multi-task fine-tuning strategy to enhance the capabilities of SQL generation models for the intrinsic alignment between SQL and text, and construct multiple generation models with distinct generation styles by fine-tuning across different SQL formats. The experimental results and comprehensive analysis demonstrate the effectiveness and robustness of our framework. Overall, XiYan-SQL achieves a new SOTA performance of 75.63% on the notable BIRD benchmark, surpassing all previous methods. It also attains SOTA performance on the Spider test set with an accuracy of 89.65%.

Paper Structure

This paper contains 20 sections, 1 equation, 5 figures, 12 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Overall Framework
  5. Schema Filter
  6. Multiple SQL Generation
  7. SQL Selection
  8. Experiments
  9. Experimental Setup
  10. Main Results
  11. Ablation Study
  12. Study of Schema Filter
  13. Study of Single SQL Generation
  14. Study of Multiple SQL Generation
  15. Study of SQL Selection
  16. ...and 5 more sections

Figures (5)

  • Figure 1: Overview of the proposed XiYan-SQL framework, including three steps: Schema Filter, Multiple SQL Generation, and SQL Selection.
  • Figure 2: Illustration of the four tasks in our Multi-Task Fine-Tuning strategy. It shows the input-output transformations for (a) the standard Text-to-SQL task, and the auxiliary tasks of (b) Question Inference, (c) Evidence Inference, and (d) Self-Refine.
  • Figure 3: Examples of "multi-format" SQL queries corresponding to the same input. Compared to the typical response (middle), the left query demonstrates a Structural Variation (e.g., using a more complex structure), while the right query illustrates a Stylistic Variation (e.g., adopting a different writing convention).
  • Figure 4: (a) Comparison of EX among different multiple candidate methods with five candidates (Figure Above). (b) Performance of multi-generator method under different candidate numbers (Figure Below).
  • Figure 5: Case study demonstrating the value of Structural Variation on a challenging query from the BIRD dev. Inputs such as evidence and the schema are omitted for clarity.