Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Models

Karime Maamari, Fadhil Abubaker, Daniel Jaroslawicz, Amine Mhedhbi

TL;DR

This paper questions the long-held assumption that schema linking is essential for accurate Text-to-SQL in the era of large language models. Through an extensive empirical study on the challenging BIRD benchmark, the authors show that when the target schema fits within the model's context window, modern LLMs can identify and leverage relevant elements without explicit linking, and that augmentation, selection, and correction techniques substantially improve end-to-end accuracy. They demonstrate that the benefit of schema linking diminishes as models become more capable, though it can still help weaker models or in settings with oversized schemas. The findings advocate maximizing the LLM's context usage and relying on non-filtering methods to preserve information, while acknowledging that real-world deployments may still require multi-stage retrieval when schemas exceed context windows.

Abstract

Schema linking is a crucial step in Text-to-SQL pipelines. Its goal is to retrieve the relevant tables and columns of a target database for a user's query while disregarding irrelevant ones. However, imperfect schema linking can often exclude required columns needed for accurate query generation. In this work, we revisit schema linking when using the latest generation of large language models (LLMs). We find empirically that newer models are adept at utilizing relevant schema elements during generation even in the presence of large numbers of irrelevant ones. As such, our Text-to-SQL pipeline entirely forgoes schema linking in cases where the schema fits within the model's context window in order to minimize issues due to filtering required schema elements. Furthermore, instead of filtering contextual information, we highlight techniques such as augmentation, selection, and correction, and adopt them to improve the accuracy of our Text-to-SQL pipeline. Our approach ranks first on the BIRD benchmark achieving an accuracy of 71.83%.

The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Models

TL;DR

This paper questions the long-held assumption that schema linking is essential for accurate Text-to-SQL in the era of large language models. Through an extensive empirical study on the challenging BIRD benchmark, the authors show that when the target schema fits within the model's context window, modern LLMs can identify and leverage relevant elements without explicit linking, and that augmentation, selection, and correction techniques substantially improve end-to-end accuracy. They demonstrate that the benefit of schema linking diminishes as models become more capable, though it can still help weaker models or in settings with oversized schemas. The findings advocate maximizing the LLM's context usage and relying on non-filtering methods to preserve information, while acknowledging that real-world deployments may still require multi-stage retrieval when schemas exceed context windows.

Abstract

Schema linking is a crucial step in Text-to-SQL pipelines. Its goal is to retrieve the relevant tables and columns of a target database for a user's query while disregarding irrelevant ones. However, imperfect schema linking can often exclude required columns needed for accurate query generation. In this work, we revisit schema linking when using the latest generation of large language models (LLMs). We find empirically that newer models are adept at utilizing relevant schema elements during generation even in the presence of large numbers of irrelevant ones. As such, our Text-to-SQL pipeline entirely forgoes schema linking in cases where the schema fits within the model's context window in order to minimize issues due to filtering required schema elements. Furthermore, instead of filtering contextual information, we highlight techniques such as augmentation, selection, and correction, and adopt them to improve the accuracy of our Text-to-SQL pipeline. Our approach ranks first on the BIRD benchmark achieving an accuracy of 71.83%.
Paper Structure (15 sections, 6 figures, 2 tables)

This paper contains 15 sections, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. Text-to-SQL Pipeline
  4. Schema Linking
  5. Experimental Setup
  6. Datasets
  7. Models
  8. Methodology
  9. Metrics
  10. Results
  11. Experiment 1: Impact of False Positives on Accuracy Given Perfect SLR
  12. Experiment 2: Impact of False Positives on Accuracy Given Actual SLR
  13. Experiment 3: Impact of Non-Filtering Stages and Techniques
  14. Discussion and Proposed Approach
  15. Conclusion

Figures (6)

  • Figure 1: A typical Text-to-SQL pipeline comparised of retrieval, generation and correction stages.
  • Figure 2: (Red) Structure of SQL Generation prompt given input query, hint, and schema; (Blue) Examples of a schema, input query, and query hint which act as contextual inputs to a prompt.
  • Figure 3: The idealized execution accuracy (IEX) as the false positive rate (FPR) varies from 99% to 0%.
  • Figure 4: Capability and sensitivity relationship.
  • Figure 6: Prompts used for schema linking. (Left) Single-Column Schema Linking (SCSL): identifying relevance of a particular column independent of the rest of the schema; (Middle + Right) Table-to-Column Schema Linking (TCSL): first identifying relevant tables then relevant columns.
  • ...and 1 more figures