JudgeSQL: Reasoning over SQL Candidates with Weighted Consensus Tournament

TL;DR

JudgeSQL addresses the test-time bottleneck in Text-to-SQL by introducing a reasoning-based SQL judge trained with verifiable rewards and a weighted consensus tournament that fuses explicit reasoning with generator confidence. The reasoning distillation stage provides structured traces, while GRPO reinforcement aligns judgments with execution correctness. The Weighted Consensus Tournament reduces search costs and enhances robustness by comparing representatives from execution-consistent sets and weighting by sample frequency. On BIRD, JudgeSQL demonstrates superior SQL judgment, strong cross-scale generalization, and robustness to generator capacity, suggesting practical utility as a reusable module for SQL selection in diverse generation pipelines.

Abstract

Text-to-SQL is a pivotal task that bridges natural language understanding and structured data access, yet it remains fundamentally challenging due to semantic ambiguity and complex compositional reasoning. While large language models (LLMs) have greatly advanced SQL generation though prompting, supervised finetuning and reinforced tuning, the shift toward test-time scaling exposes a new bottleneck: selecting the correct query from a diverse candidate pool. Existing selection approaches, such as self-consistency or best-of-$N$ decoding, provide only shallow signals, making them prone to inconsistent scoring, fragile reasoning chains, and a failure to capture fine-grained semantic distinctions between closely related SQL candidates. To this end, we introduce JudgeSQL, a principled framework that redefines SQL candidate selection through structured reasoning and weighted consensus tournament mechanism. JudgeSQL develops a reasoning-based SQL judge model that distills reasoning traces with reinforcement learning guided by verifiable rewards, enabling accurate and interpretable judgments. Building on this, a weighted consensus tournament integrates explicit reasoning preferences with implicit generator confidence, yielding selections that are both more reliable and more efficient. Extensive experiments on the BIRD benchmark demonstrate that JudgeSQL exhibits superior SQL judgment capabilities and good cross-scale generalization and robustness to generator capacity.

Figures (5)

• Figure 1: Trends of EX scores for greedy decoding, self-consistency, and Pass@k methods across different sampling sizes on the BIRD dev set.
• Figure 2: Overview of the JudgeSQL framework.
• Figure 3: Performance by query difficulty on BIRD dev set.
• Figure 4: The task description of the example case study.
• Figure 5: The inference process of GRPO model for this case study.