FuncEvalGMN: Evaluating Functional Correctness of SQL via Graph Matching Network

TL;DR

FuncEvalGMN tackles the challenge of evaluating functional correctness in NL2SQL by framing SQL queries as Relational Operator Trees (RelNode) and encoding them into program graphs that capture both syntax and semantics. The method uses a Graph Matching Network with global positional encoding to measure cross-graph similarity, providing a robust alternative to binary execution scores and lexical matching metrics. A rule-based baseline (RelPM) and the Spider-Pair dataset are introduced to benchmark functional equivalence in SQL generation. Empirical results show that FuncEvalGMN achieves state-of-the-art performance on Spider-Pair, generalizes to the BIRD NL2SQL dataset, and offers a data-efficient means to assess SQL correctness beyond execution-based evaluation.

Abstract

In this paper, we propose a novel graph-based methodology to evaluate the functional correctness of SQL generation. Conventional metrics for assessing SQL code generation, such as matching-based and execution-based methods (e.g., exact set match and execution accuracy), are subject to two primary limitations. Firstly, the former fails to effectively assess functional correctness, as different SQL queries may possess identical functionalities. Secondly, the latter is susceptible to producing false positive samples in evaluations. Our proposed evaluation method, \texttt{FuncEvalGMN}, does not depend on the sufficient preparation of the test data, and it enables precise testing of the functional correctness of the code. Firstly, we parse SQL using a relational operator tree (ROT) called \textit{Relnode}, which contains rich semantic information from the perspective of logical execution.Then, we introduce a GNN-based approach for predicting the functional correctness of generated SQL. This approach incorporates global positional embeddings to address the limitations with the loss of topological information in conventional graph matching frameworks. As an auxiliary contribution, we propose a rule-based matching algorithm, Relnode Partial Matching (\texttt{RelPM}) as a baseline. Finally, we contribute a dataset, \texttt{Pair-Aug-Spider} with a training set and two testing sets, each comprising pairs of SQL codes to simulate various SQL code evaluation scenarios. The training set and one testing dataset focus on code generation using large language models (LLMs), while the other emphasizes SQL equivalence rewriting.

Figures (15)

• Figure 1: Overview of FuncGMN approach
• Figure 2: SQL Representations using RelNode and AST. The AST purely abstracts SQL from a syntactic perspective, whereas RelNode provides more semantic information from an execution standpoint. In RelNode, orange edges represent logic flows, connected at the second layer of the RelNode, indicating the logical sequence of clause execution. Green-colored data flows represents the pathways of data across various clauses, associating with nodes representing column names.
• Figure 3: Positional Encoding.
• Figure 4: Graph Embedding.
• Figure 5: Partial Matching