Enhancing SQL Query Generation with Neurosymbolic Reasoning

TL;DR

The paper tackles text-to-SQL-with-examples by introducing Xander, a neurosymbolic architecture that couples pretrained LMs with symbolic and neural checkers and a repair module to navigate a solution tree via Best-First Search. It introduces Normalized SQL to reduce stylistic variation and enable earlier error detection. Empirical results on the Spider dataset show that Xander improves execution accuracy by $10.9\%$ and reduces runtime by $28\%$ across several open-source LMs, enabling smaller models to outperform larger ones. The work demonstrates that neurosymbolic reasoning can be an effective alternative to scaling model size for code generation tasks.

Abstract

Neurosymbolic approaches blend the effectiveness of symbolic reasoning with the flexibility of neural networks. In this work, we propose a neurosymbolic architecture for generating SQL queries that builds and explores a solution tree using Best-First Search, with the possibility of backtracking. For this purpose, it integrates a Language Model (LM) with symbolic modules that help catch and correct errors made by the LM on SQL queries, as well as guiding the exploration of the solution tree. We focus on improving the performance of smaller open-source LMs, and we find that our tool, Xander, increases accuracy by an average of 10.9% and reduces runtime by an average of 28% compared to the LM without Xander, enabling a smaller LM (with Xander) to outperform its four-times larger counterpart (without Xander).

Figures (8)

• Figure 1: Illustration of the problem definition: given $(\mathcal{D},q,o)$ find $\ell$.
• Figure 2: Xander overview. During inference Xander consists of three main modules connected by Best First Search (BFS). BFS explores the solution tree by choosing the most promising incomplete query (yellow box) and passing it to the LM. The LM then provides five candidates that extend the query by a single token. A Partial-Query Checker dismisses candidate queries containing errors (red box). The Partial-Query Checker passes queries without errors back to BFS if they are incomplete or to Query Test-and-Repair if they are complete. Finally, Query Test-and-Repair module executes the full query and verifies that it executes to the correct result, in which case it is returned to the user (green box). If it does not, they are dismissed (red box) and BFS backtracks.
• Figure 3: SQL (left) Normalized SQL (right). SQL uses an alias which is highlighted in red. Normalized SQL replaces the alias with what it points to (in this example, t1 is replaced with singer).
• Figure 4: Illustration of how Normalized SQL makes learning easier. Notice how in (unnormalized) SQL case, despite a minor change to the input specification, a drastic change is observed in the output's byte-pair encoding.
• Figure 5: Illustration of three different types of errors. Notice how example error query is syntactically correct and executes, but the tuples brought back do not contain the tuple provided by the user.