PET-SQL: A Prompt-Enhanced Two-Round Refinement of Text-to-SQL with Cross-consistency

TL;DR

PET-SQL introduces a prompting framework that first generates a preliminary SQL (PreSQL) to anchor schema linking, then produces a refined final SQL (FinSQL) with a simplified prompt guided by linked schema. It adds a cross-consistency post-refinement across multiple LLMs at low temperature, outperforming prior methods on Spider and generalizing across several foundation LLMs. Key innovations include reference-enhanced prompts with cell-value references and foreign-key declarations, a PreSQL-based schema linking pipeline, and difficulty-aware voting. The approach achieves 87.6% execution accuracy on Spider, advancing open-source Text2SQL performance and demonstrating how prompt design and cross-model ensemble can surpass single-model gains without model fine-tuning.

Abstract

Recent advancements in Text-to-SQL (Text2SQL) emphasize stimulating the large language models (LLM) on in-context learning, achieving significant results. Nevertheless, they face challenges when dealing with verbose database information and complex user intentions. This paper presents a two-stage framework to enhance the performance of current LLM-based natural language to SQL systems. We first introduce a novel prompt representation, called reference-enhanced representation, which includes schema information and randomly sampled cell values from tables to instruct LLMs in generating SQL queries. Then, in the first stage, question-SQL pairs are retrieved as few-shot demonstrations, prompting the LLM to generate a preliminary SQL (PreSQL). After that, the mentioned entities in PreSQL are parsed to conduct schema linking, which can significantly compact the useful information. In the second stage, with the linked schema, we simplify the prompt's schema information and instruct the LLM to produce the final SQL. Finally, as the post-refinement module, we propose using cross-consistency across different LLMs rather than self-consistency within a particular LLM. Our methods achieve new SOTA results on the Spider benchmark, with an execution accuracy of 87.6%.

Figures (5)

• Figure 1: (a) A three-stage paradigm of SOTA Text2SQL. (b) We summarize the distribution of error types in DIN-SQL din-sql. We elaborate on the methods for each type: PreSQL-based schema linking for errors in schema linking, cell values references-enhanced prompt for incorrect cell values, and cross-consistency for other errors caused by typos or syntax mistakes. (c) Existing schema linking directly lets LLM guess the schema, we let LLM first generate a preliminary SQL and parse table/column entities from PreSQL, increasing accuracy by +4%. (d) Our proposed cross-consistency enables the LLMs to operate at low temperatures, thereby reducing hallucination.
• Figure 1: OpenAI Demonstration v.s. Code Representation.
• Figure 2: The overview of PET-SQL framework. It first elaborates on an effective SQL-Tailored prompt (Sec. \ref{['sec:prompt']}), which can be enhanced with selected demos as few-shot in-context learning. Then, PreSQL is generated by LLMs for parsing linked schema (Sec. \ref{['sec:schema']}), where linked entities are retained while unrelated parts are removed from the prompt, thereby simplifying it. After that, the Final SQL can be obtained from LLMs with the concise prompt (Sec. \ref{['sec:finsql']}). Finally, cross-consistency is proposed to post-refine the results generated from multiple LLMs (Sec. \ref{['sec:cc']}). Our PreSQL-based schema linking shortens the prompt length by 32%, and our cross-consistency boosts the Text2SQL accuracy by 2%.
• Figure 2: The proposed SQL-Tailored prompt, containing OR, CV, and FK. It is zero-shot without demos.
• Figure 3: Three question-SQL pairs are retrieved and prefixed as a 3-shot prompt, followed by Listing \ref{['list:prompt']} to serve as the full prompt feeding into LLM for PreSQL.