Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

REFORMER: A ChatGPT-Driven Data Synthesis Framework Elevating Text-to-SQL Models

Shenyang Liu, Saleh Almohaimeed, Liqiang Wang

TL;DR

Text-to-SQL models suffer from limited domain data, hindering generalization. REFORMER introduces a ChatGPT-driven retrieve-and-edit framework to synthesize domain-adapted (question, SQL) pairs without fine-tuning, backed by cycle-consistency validation and two paraphrasing methods. Empirical results show REFORMER improves EM and EX over prior data augmentation approaches across Spider-domain categories, highlighting the practical impact of LLM-assisted data synthesis for Text-to-SQL. The work also analyzes data quality/diversity, error modes, and the trade-offs of paraphrasing strategies, pointing to future work on automatic prompts, additional LLMs, and broader datasets.

Abstract

The existing Text-to-SQL models suffer from a shortage of training data, inhibiting their ability to fully facilitate the applications of SQL queries in new domains. To address this challenge, various data synthesis techniques have been employed to generate more diverse and higher quality data. In this paper, we propose REFORMER, a framework that leverages ChatGPT's prowess without the need for additional training, to facilitate the synthesis of (question, SQL query) pairs tailored to new domains. Our data augmentation approach is based on a "retrieve-and-edit" method, where we generate new questions by filling masked question using explanation of SQL queries with the help of ChatGPT. Furthermore, we demonstrate that cycle consistency remains a valuable method of validation when applied appropriately. Our experimental results show that REFORMER consistently outperforms previous data augmentation methods. To further investigate the power of ChatGPT and create a general data augmentation method, we also generate the new data by paraphrasing the question in the dataset and by paraphrasing the description of a new SQL query that is generated by ChatGPT as well. Our results affirm that paraphrasing questions generated by ChatGPT help augment the original data.

REFORMER: A ChatGPT-Driven Data Synthesis Framework Elevating Text-to-SQL Models

TL;DR

Text-to-SQL models suffer from limited domain data, hindering generalization. REFORMER introduces a ChatGPT-driven retrieve-and-edit framework to synthesize domain-adapted (question, SQL) pairs without fine-tuning, backed by cycle-consistency validation and two paraphrasing methods. Empirical results show REFORMER improves EM and EX over prior data augmentation approaches across Spider-domain categories, highlighting the practical impact of LLM-assisted data synthesis for Text-to-SQL. The work also analyzes data quality/diversity, error modes, and the trade-offs of paraphrasing strategies, pointing to future work on automatic prompts, additional LLMs, and broader datasets.

Abstract

The existing Text-to-SQL models suffer from a shortage of training data, inhibiting their ability to fully facilitate the applications of SQL queries in new domains. To address this challenge, various data synthesis techniques have been employed to generate more diverse and higher quality data. In this paper, we propose REFORMER, a framework that leverages ChatGPT's prowess without the need for additional training, to facilitate the synthesis of (question, SQL query) pairs tailored to new domains. Our data augmentation approach is based on a "retrieve-and-edit" method, where we generate new questions by filling masked question using explanation of SQL queries with the help of ChatGPT. Furthermore, we demonstrate that cycle consistency remains a valuable method of validation when applied appropriately. Our experimental results show that REFORMER consistently outperforms previous data augmentation methods. To further investigate the power of ChatGPT and create a general data augmentation method, we also generate the new data by paraphrasing the question in the dataset and by paraphrasing the description of a new SQL query that is generated by ChatGPT as well. Our results affirm that paraphrasing questions generated by ChatGPT help augment the original data.

Paper Structure

This paper contains 27 sections, 3 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. SQL-to-Text Task
  4. Large Language Models and Prompt Engineering For SQL-to-Text
  5. Data synthesis with REFORMER
  6. Related SQL queries and question templates preparation
  7. New Question Generation
  8. Result validation
  9. Paraphrasing-Based Data Synthesis Method Using ChatGPT
  10. Direct Paraphrasing with Schema Information
  11. Paraphrasing Using the Description of Crafted SQL Queries
  12. Experiment Setting
  13. Datasets
  14. Text-to-SQL Model
  15. SQL-to-Text Model
  16. ...and 12 more sections

Figures (3)

  • Figure 1: Framework for REFORMER and Paraphrasing-based Data Sythesis methods
  • Figure 2: Prompts for data generation, where "query", "query_similar", "explanation 1/2", "question_result", "question_template" etc. denote the corresponding placeholders, which will be replaced by real queries or explanations when handing SQL query and question pairs.
  • Figure 3: Errors from REFORMER