Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Generalization in Semantic Parsing by Increasing Natural Language Variation

Irina Saparina, Mirella Lapata

TL;DR

This work addresses the limited linguistic variation in Spider-based text-to-SQL training, which hampers cross-domain generalization. It proposes using large language models to generate diverse, natural reformulations of Spider questions, substantially increasing training data. Across multiple parsers, augmented training improves robustness to NLQ perturbations and zero-shot generalization to GeoQuery and KaggleDBQA, confirming the value of broad NL rewrites. The approach demonstrates that prompt-driven LLM rewrites can meaningfully enhance semantic parsing performance, while acknowledging limitations around SQL/DB-schema variations and reliance on a black-box LLM.

Abstract

Text-to-SQL semantic parsing has made significant progress in recent years, with various models demonstrating impressive performance on the challenging Spider benchmark. However, it has also been shown that these models often struggle to generalize even when faced with small perturbations of previously (accurately) parsed expressions. This is mainly due to the linguistic form of questions in Spider which are overly specific, unnatural, and display limited variation. In this work, we use data augmentation to enhance the robustness of text-to-SQL parsers against natural language variations. Existing approaches generate question reformulations either via models trained on Spider or only introduce local changes. In contrast, we leverage the capabilities of large language models to generate more realistic and diverse questions. Using only a few prompts, we achieve a two-fold increase in the number of questions in Spider. Training on this augmented dataset yields substantial improvements on a range of evaluation sets, including robustness benchmarks and out-of-domain data.

Improving Generalization in Semantic Parsing by Increasing Natural Language Variation

TL;DR

This work addresses the limited linguistic variation in Spider-based text-to-SQL training, which hampers cross-domain generalization. It proposes using large language models to generate diverse, natural reformulations of Spider questions, substantially increasing training data. Across multiple parsers, augmented training improves robustness to NLQ perturbations and zero-shot generalization to GeoQuery and KaggleDBQA, confirming the value of broad NL rewrites. The approach demonstrates that prompt-driven LLM rewrites can meaningfully enhance semantic parsing performance, while acknowledging limitations around SQL/DB-schema variations and reliance on a black-box LLM.

Abstract

Text-to-SQL semantic parsing has made significant progress in recent years, with various models demonstrating impressive performance on the challenging Spider benchmark. However, it has also been shown that these models often struggle to generalize even when faced with small perturbations of previously (accurately) parsed expressions. This is mainly due to the linguistic form of questions in Spider which are overly specific, unnatural, and display limited variation. In this work, we use data augmentation to enhance the robustness of text-to-SQL parsers against natural language variations. Existing approaches generate question reformulations either via models trained on Spider or only introduce local changes. In contrast, we leverage the capabilities of large language models to generate more realistic and diverse questions. Using only a few prompts, we achieve a two-fold increase in the number of questions in Spider. Training on this augmented dataset yields substantial improvements on a range of evaluation sets, including robustness benchmarks and out-of-domain data.
Paper Structure (29 sections, 2 figures, 8 tables)

This paper contains 29 sections, 2 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Out-of-domain Generalization
  4. Robustness to Perturbations
  5. Data Augmentation
  6. Motivation
  7. Problem Formulation
  8. Types of Utterances in Semantic Parsing
  9. Utterances which demonstrate prior knowledge
  10. Utterances which do not demonstrate prior knowledge
  11. Utterances which demonstrate knowledge of the database schema
  12. Data Generation
  13. Experimental Setup
  14. Training Datasets
  15. Evaluation Datasets
  16. ...and 14 more sections

Figures (2)

  • Figure 1: Different types of questions that are related to the same database (only relevant tables and columns are shown) and map to the same SQL query.
  • Figure 2: Distribution of cosine similarities between Spider questions and generated reformulations.