Is Long Context All You Need? Leveraging LLM's Extended Context for NL2SQL

TL;DR

NL2SQL must resolve natural-language ambiguity against complex database schemas. The authors study long-context LLMs (Gemini-1.5-pro) in a three-stage pipeline (Generate, Fix & Rewrite, Verify) and incorporate full schemas, descriptions, hints, and synthetic examples into the extended context, without model fine-tuning. They demonstrate competitive accuracy across BIRD, SPIDER, KaggleDBQA, and BEAVER benchmarks, and show that including all tables improves recall, albeit with latency costs. The work provides practical guidance on deploying long-context NL2SQL, balancing recall and latency, and suggests directions such as knowledge-graph integration for future improvements.

Abstract

Large Language Models (LLMs) have demonstrated impressive capabilities across a range of natural language processing tasks. In particular, improvements in reasoning abilities and the expansion of context windows have opened new avenues for leveraging these powerful models. NL2SQL is challenging in that the natural language question is inherently ambiguous, while the SQL generation requires a precise understanding of complex data schema and semantics. One approach to this semantic ambiguous problem is to provide more and sufficient contextual information. In this work, we explore the performance and the latency trade-offs of the extended context window (a.k.a., long context) offered by Google's state-of-the-art LLM (\textit{gemini-1.5-pro}). We study the impact of various contextual information, including column example values, question and SQL query pairs, user-provided hints, SQL documentation, and schema. To the best of our knowledge, this is the first work to study how the extended context window and extra contextual information can help NL2SQL generation with respect to both accuracy and latency cost. We show that long context LLMs are robust and do not get lost in the extended contextual information. Additionally, our long-context NL2SQL pipeline based on Google's \textit{gemini-pro-1.5} achieve strong performances on various benchmark datasets without finetuning and expensive self-consistency based techniques.

Figures (6)

• Figure 1: Long Context NL2SQL Pipeline. Leveraging long context LLMs can make the retrieval step less critical and the agentic workflow (generate$\rightarrow$fix & rewrite$\rightarrow$verify) more accurate with extra contextual information.
• Figure 2: Top K relevant table retrieval simulation. The top-K relevant tables are selected from BIRD dev based on question embedding similarity, without a specific target DB, as supported in some production RDBMSs.
• Figure 3: A hint (Evidence) prescribing the nuanced column reference and the mathematical expression needed to answer a challenging question from BIRD dev.
• Figure 4: Question similarity-based example selection ($\sigma$) vs. online example generation. The baseline uses no examples; examples were retrieved from BIRD train dataset, dev dataset (excluding the GT) and synthetic generated examples. train + GT uses the retrieved train examples and the ground-truth SQL from dev.
• Figure 5: Single LLM request latency by different context sizes. The axes are in log-scale, and there is a step-wise linear relationship between latency and context size. Both pro and flash models suffer from increased and high-variance latency per request beyond $>32k$ tokens context size.