Text-to-SQL based on Large Language Models and Database Keyword Search

TL;DR

This work tackles the gap between benchmark Text-to-SQL performance and real-world deployment on large, complex relational schemas. It introduces a two-module strategy—schema linking and SQL query compilation—augmented by a database keyword search platform called DANKE and dynamic few-shot prompting, including a synthesized view $V$ that captures necessary joins. The key contributions are the integration of DANKE's Keyword Match and View Synthesis services with a synthetic data generation pipeline, and empirical evidence that the method exceeds state-of-the-art baselines on a realistic 100-question benchmark, achieving high accuracy and recall. The approach has practical impact for production NL interfaces to databases, and future work includes broader evaluations, disambiguation loops, and scalability improvements.

Abstract

Text-to-SQL prompt strategies based on Large Language Models (LLMs) achieve remarkable performance on well-known benchmarks. However, when applied to real-world databases, their performance is significantly less than for these benchmarks, especially for Natural Language (NL) questions requiring complex filters and joins to be processed. This paper then proposes a strategy to compile NL questions into SQL queries that incorporates a dynamic few-shot examples strategy and leverages the services provided by a database keyword search (KwS) platform. The paper details how the precision and recall of the schema-linking process are improved with the help of the examples provided and the keyword-matching service that the KwS platform offers. Then, it shows how the KwS platform can be used to synthesize a view that captures the joins required to process an input NL question and thereby simplify the SQL query compilation step. The paper includes experiments with a real-world relational database to assess the performance of the proposed strategy. The experiments suggest that the strategy achieves an accuracy on the real-world relational database that surpasses state-of-the-art approaches. The paper concludes by discussing the results obtained.

Figures (4)

• Figure 1: Proposed strategy.
• Figure 2: Schema linking module.
• Figure 3: SQL query compilation module.
• Figure 4: The referential dependencies diagram of a simplified and anonymized version of the industrial database schema.