SQLSpace: A Representation Space for Text-to-SQL to Discover and Mitigate Robustness Gaps

TL;DR

SQLSpace provides a compact, human-interpretable representation for NL2SQL examples by extracting 187 binary predicates from five aspect-based descriptions, enabling fine-grained benchmarking and model analysis beyond aggregate accuracy. The framework combines a semi-automatic description-generation step, predicate discovery, deduplication, and binary vector construction, producing representations that reveal benchmark composition, model blind spots, and potential for correctness-guided query rewriting. Through analyses of Spider-Dev, Bird-Dev, and Spider-Realistic, SQLSpace demonstrates how cluster-level and dataset-level insights can inform model selection, robustness benchmarking, and targeted data augmentation, with practical cost considerations and potential for online rewriting via a correctness estimator. The work highlights how interpretable representations can uncover systematic weaknesses, guide efficient model deployment, and motivate future improvements in NL2SQL robustness research and benchmark design.

Abstract

We introduce SQLSpace, a human-interpretable, generalizable, compact representation for text-to-SQL examples derived with minimal human intervention. We demonstrate the utility of these representations in evaluation with three use cases: (i) closely comparing and contrasting the composition of popular text-to-SQL benchmarks to identify unique dimensions of examples they evaluate, (ii) understanding model performance at a granular level beyond overall accuracy scores, and (iii) improving model performance through targeted query rewriting based on learned correctness estimation. We show that SQLSpace enables analysis that would be difficult with raw examples alone: it reveals compositional differences between benchmarks, exposes performance patterns obscured by accuracy alone, and supports modeling of query success.

Figures (7)

• Figure 1: Our framework generates compact representations of NL2SQL examples by ingesting a dataset, discovering shared properties of dataset items in natural language, and evaluating these properties on examples to produce binary feature vectors. Clustering these feature vectors and examining a model's cluster-level accuracy reveals classes of examples that it systematically struggles with, called blind spots.
• Figure 2: Aggregate accuracy may obscure important performance characteristics of models. Three models with identical 80% accuracy on benchmark $B$ exhibit different error patterns across example classes, while varying class distributions across benchmarks can explain performance differences.
• Figure 3: We discover representations of NL2SQL examples, and then use these representations in two applications: fine-grained analysis of models and benchmarks, and improving performance of models by rewriting NL questions to eliminate features associated with incorrect predictions.
• Figure 4: Visualizing a UMAP projection (left) of our example representations for three NL2SQL datasets reveals classes of examples across datasets that share certain properties. Computing the proportions of examples exhibiting certain features (right) reveals dimensions along which the composition of datasets statistically significantly differs.
• Figure 5: UMAP projection of feature vectors from Bird-Dev and Spider-Dev. We color each point in Bird-Dev using the hand-annotated metadata released with the dataset. We observe that areas of overlap between Bird-Dev and Spider-Dev typically occur on examples annotated as "simple" in Bird-Dev.