Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

You Only Read Once (YORO): Learning to Internalize Database Knowledge for Text-to-SQL

Hideo Kobayashi, Wuwei Lan, Peng Shi, Shuaichen Chang, Jiang Guo, Henghui Zhu, Zhiguo Wang, Patrick Ng

TL;DR

This work tackles inefficiency and knowledge gaps in text-to-SQL by eliminating per-question schema encoding at inference. It introduces YORO, a paradigm that internalizes database knowledge through a knowledge acquisition phase where domain-expert models are fine-tuned on synthetic NLQ-SQL data, enabling inference without access to schema or value retrievers. The approach uses a lean prompt design that retains only a database ID, and leverages skeleton-based SQL synthesis, SQL/NLQ generation, and in-context learning to create synthetic data, culminating in per-database expert models. Evaluations on Spider, KaggleDBQA, and BIRD show YORO achieves competitive accuracy, with pronounced gains on large databases and in challenging value retrieval scenarios, while substantially reducing input length and inference costs. The work also demonstrates the viability of parameter-efficient fine-tuning (LoRA) and provides insights from extensive ablations and case studies, with plans to publicly release code and data.

Abstract

While significant progress has been made on the text-to-SQL task, recent solutions repeatedly encode the same database schema for every question, resulting in unnecessary high inference cost and often overlooking crucial database knowledge. To address these issues, we propose You Only Read Once (YORO), a novel paradigm that directly internalizes database knowledge into the parametric knowledge of a text-to-SQL model during training and eliminates the need for schema encoding during inference. YORO significantly reduces the input token length by 66%-98%. Despite its shorter inputs, our empirical results demonstrate YORO's competitive performances with traditional systems on three benchmarks as well as its significant outperformance on large databases. Furthermore, YORO excels in handling questions with challenging value retrievals such as abbreviation.

You Only Read Once (YORO): Learning to Internalize Database Knowledge for Text-to-SQL

TL;DR

This work tackles inefficiency and knowledge gaps in text-to-SQL by eliminating per-question schema encoding at inference. It introduces YORO, a paradigm that internalizes database knowledge through a knowledge acquisition phase where domain-expert models are fine-tuned on synthetic NLQ-SQL data, enabling inference without access to schema or value retrievers. The approach uses a lean prompt design that retains only a database ID, and leverages skeleton-based SQL synthesis, SQL/NLQ generation, and in-context learning to create synthetic data, culminating in per-database expert models. Evaluations on Spider, KaggleDBQA, and BIRD show YORO achieves competitive accuracy, with pronounced gains on large databases and in challenging value retrieval scenarios, while substantially reducing input length and inference costs. The work also demonstrates the viability of parameter-efficient fine-tuning (LoRA) and provides insights from extensive ablations and case studies, with plans to publicly release code and data.

Abstract

While significant progress has been made on the text-to-SQL task, recent solutions repeatedly encode the same database schema for every question, resulting in unnecessary high inference cost and often overlooking crucial database knowledge. To address these issues, we propose You Only Read Once (YORO), a novel paradigm that directly internalizes database knowledge into the parametric knowledge of a text-to-SQL model during training and eliminates the need for schema encoding during inference. YORO significantly reduces the input token length by 66%-98%. Despite its shorter inputs, our empirical results demonstrate YORO's competitive performances with traditional systems on three benchmarks as well as its significant outperformance on large databases. Furthermore, YORO excels in handling questions with challenging value retrievals such as abbreviation.
Paper Structure (32 sections, 5 figures, 6 tables)

This paper contains 32 sections, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Text-to-SQL.
  4. Context Compression.
  5. Synthesizing data for Text-to-SQL.
  6. YORO
  7. Prompt Structure
  8. Database Knowledge Acquisition
  9. SQL Skeleton Extraction.
  10. SQL Generation.
  11. NLQ Generation.
  12. Domain Experts
  13. Evaluation
  14. Experimental Setup
  15. Evaluation datasets.
  16. ...and 17 more sections

Figures (5)

  • Figure 1: Comparison of traditional method and YORO.
  • Figure 2: Overview of YORO. YORO comprehends and internalizes database knowledge through fine-tuning text-to-SQL expert models on synthetic NLQ and SQL data. Comparing with traditional methods, it leads to significantly shorter inputs and does not rely on the value retrieval step.
  • Figure 3: Examples of different prompts for the same data in Spider Dev, each followed by the same NLQ.
  • Figure 4: Performance of YORO trained with varying amounts of synthetic data using Mistral-7B.
  • Figure 5: Examples of challenging value retrieval scenarios. The value retriever finds no value in these examples.