Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers

Chia-Hsuan Lee, Oleksandr Polozov, Matthew Richardson

TL;DR

KaggleDBQA introduces a realistic cross-domain text-to-SQL evaluation framework built on real Kaggle databases with domain-specific data and free-form questions, augmented by in-domain database documentation. The work argues that zero-shot benchmarks inadequately reflect deployment and proposes a few-shot regime (≈30% in-domain data) plus documentation to close the gap. Baseline results show state-of-the-art parsers struggle on KaggleDBQA, but incorporating in-domain descriptions and adaptation nearly doubles accuracy from $13.56\%$ to $26.77\%$, and normalization helps further. The dataset and methodology aim to bridge academia and industry by stressing practical schema linking and knowledge usage, with significant implications for deploying Text-to-SQL in real applications.

Abstract

The goal of database question answering is to enable natural language querying of real-life relational databases in diverse application domains. Recently, large-scale datasets such as Spider and WikiSQL facilitated novel modeling techniques for text-to-SQL parsing, improving zero-shot generalization to unseen databases. In this work, we examine the challenges that still prevent these techniques from practical deployment. First, we present KaggleDBQA, a new cross-domain evaluation dataset of real Web databases, with domain-specific data types, original formatting, and unrestricted questions. Second, we re-examine the choice of evaluation tasks for text-to-SQL parsers as applied in real-life settings. Finally, we augment our in-domain evaluation task with database documentation, a naturally occurring source of implicit domain knowledge. We show that KaggleDBQA presents a challenge to state-of-the-art zero-shot parsers but a more realistic evaluation setting and creative use of associated database documentation boosts their accuracy by over 13.2%, doubling their performance.

KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers

TL;DR

KaggleDBQA introduces a realistic cross-domain text-to-SQL evaluation framework built on real Kaggle databases with domain-specific data and free-form questions, augmented by in-domain database documentation. The work argues that zero-shot benchmarks inadequately reflect deployment and proposes a few-shot regime (≈30% in-domain data) plus documentation to close the gap. Baseline results show state-of-the-art parsers struggle on KaggleDBQA, but incorporating in-domain descriptions and adaptation nearly doubles accuracy from to , and normalization helps further. The dataset and methodology aim to bridge academia and industry by stressing practical schema linking and knowledge usage, with significant implications for deploying Text-to-SQL in real applications.

Abstract

The goal of database question answering is to enable natural language querying of real-life relational databases in diverse application domains. Recently, large-scale datasets such as Spider and WikiSQL facilitated novel modeling techniques for text-to-SQL parsing, improving zero-shot generalization to unseen databases. In this work, we examine the challenges that still prevent these techniques from practical deployment. First, we present KaggleDBQA, a new cross-domain evaluation dataset of real Web databases, with domain-specific data types, original formatting, and unrestricted questions. Second, we re-examine the choice of evaluation tasks for text-to-SQL parsers as applied in real-life settings. Finally, we augment our in-domain evaluation task with database documentation, a naturally occurring source of implicit domain knowledge. We show that KaggleDBQA presents a challenge to state-of-the-art zero-shot parsers but a more realistic evaluation setting and creative use of associated database documentation boosts their accuracy by over 13.2%, doubling their performance.

Paper Structure

This paper contains 33 sections, 2 figures, 12 tables.

Table of Contents

  1. Introduction
  2. KaggleDBQA
  3. Related Work
  4. Text-to-SQL Semantic Parsing
  5. Few-shot learning
  6. Generalization and Practical usability
  7. Leveraging other resources for learning
  8. KaggleDBQA: A Real World Dataset
  9. Database Collection
  10. Questions
  11. Database Documentation
  12. Few-shot Evaluation Setting
  13. Dataset Statistics and Comparison
  14. Experiments
  15. Baseline Results
  16. ...and 18 more sections

Figures (2)

  • Figure 1: Two table excerpts from the Student Math Score database in KaggleDBQA and an example question-SQL pair. The column names are abbreviated (e.g.) or obscure (e.g., ) but documentation (e.g. column descriptions) alleviates this. Source: https://kaggle.com/loganhenslee/studentmathscores.
  • Figure 2: Comparisons of text-to-SQL datasets in terms of SQL structure hardness. KaggleDBQA has more complex SQL query structure than the Spider dev set.