End-to-end Text-to-SQL Generation within an Analytics Insight Engine

TL;DR

The paper tackles the challenge of generating highly complex SQL under strict latency and domain-specific terminology constraints by presenting an end-to-end Text-to-SQL pipeline powered by large language models. It introduces a pre-processing phase that builds an external knowledge set (examples, instructions, and schema), a retrieval-generation-self-correction inference loop, and an adaptation mechanism that updates knowledge based on user and execution feedback. Key contributions include a hierarchically decomposed, CTE-based representation of queries, CoT-based generation augmented with pseudo-SQL, and execution-guided self-correction that enables continual improvement. The approach aims to deliver robust, low-latency SQL generation suitable for enterprise analytics, with explicit mechanisms for bootstrapping in new environments and evolving knowledge to reflect domain practices.

Abstract

Recent advancements in Text-to-SQL have pushed database management systems towards greater democratization of data access. Today's language models are at the core of these advancements. They enable impressive Text-to-SQL generation as experienced in the development of Distyl AI's Analytics Insight Engine. Its early deployment with enterprise customers has highlighted three core challenges. First, data analysts expect support with authoring SQL queries of very high complexity. Second, requests are ad-hoc and, as such, require low latency. Finally, generation requires an understanding of domain-specific terminology and practices. The design and implementation of our Text-to-SQL generation pipeline, powered by large language models, tackles these challenges. The core tenants of our approach rely on external knowledge that we extract in a pre-processing phase, on retrieving the appropriate external knowledge at query generation time, and on decomposing SQL query generation following a hierarchical CTE-based structure. Finally, an adaptation framework leverages feedback to update the external knowledge, in turn improving query generation over time. We give an overview of our end-to-end approach and highlight the operators generating SQL during inference.

Figures (1)

• Figure 1: Overview of Text-to-SQL generation pipeline with 3 steps: i) external knowledge retrieval to add context; ii) LLM SQL generation task; and iii) self-correction, which re-generates based on syntactic and semantic feedback from execution.