Reliable Text-to-SQL with Adaptive Abstention

TL;DR

The paper addresses unreliability in text-to-SQL systems driven by natural language ambiguity and limited context. It introduces Reliable Text-to-SQL (RTS), a framework that emphasizes the schema linking phase and incorporates abstention and human-in-the-loop mechanisms, underpinned by a Branching Point Predictor (BPP) that uses conformal prediction to provide probabilistic guarantees. RTS demonstrates near-perfect schema linking on the BIRD benchmark, enabling a small, non-SOTA SQL generator to closely approach state-of-the-art accuracy when aided by human input. The work highlights the value of transparent-box LLMs combined with human feedback for robust database interfaces and provides practical mechanisms (surrogate filters, abstention, and human-in-the-loop prompts) to improve reliability in real-world deployments. Overall, RTS offers a scalable path to safer NL-to-SQL systems by quantifying uncertainty, abstaining when needed, and incorporating expert input to maintain high accuracy with lightweight models.

Abstract

Large language models (LLMs) have revolutionized natural language interfaces for databases, particularly in text-to-SQL conversion. However, current approaches often generate unreliable outputs when faced with ambiguity or insufficient context. We present Reliable Text-to-SQL (RTS), a novel framework that enhances query generation reliability by incorporating abstention and human-in-the-loop mechanisms. RTS focuses on the critical schema linking phase, which aims to identify the key database elements needed for generating SQL queries. It autonomously detects potential errors during the answer generation process and responds by either abstaining or engaging in user interaction. A vital component of RTS is the Branching Point Prediction (BPP) which utilizes statistical conformal techniques on the hidden layers of the LLM model for schema linking, providing probabilistic guarantees on schema linking accuracy. We validate our approach through comprehensive experiments on the BIRD benchmark, demonstrating significant improvements in robustness and reliability. Our findings highlight the potential of combining transparent-box LLMs with human-in-the-loop processes to create more robust natural language interfaces for databases. For the BIRD benchmark, our approach achieves near-perfect schema linking accuracy, autonomously involving a human when needed. Combined with query generation, we demonstrate that near-perfect schema linking and a small query generation model can almost match SOTA accuracy achieved with a model orders of magnitude larger than the one we use.

Figures (7)

• Figure 1: Example of Error-prone Text-to-SQL in BIRD Benchmark
• Figure 2: Example of the RTS Framework: When probable errors are detected during answer generation, the model will either abstain from generating the query (left), ask the user for confirmation of the potential schema (middle), or prompt the user for hints (right).
• Figure 3: Statistic For Finetuned Deepseek Model on the BIRD Development Dataset
• Figure 4: Branching Points: During the generation of schema linking tokens, we compare the predicted token with ground truth. If the prediction diverges from ground truth, as in the prediction $\hat{x_3}$ above, teacher forcing will provide the correct token $x_3$ to the input to continue generation.
• Figure 5: Branching Point Predictor: Calibrated Classifiers per layer are trained to predict branching points. We aggregate the predictions in a principled manner to produce a final prediction.

Theorems & Definitions (3)

• Theorem 1
• Theorem 2
• Theorem 3