Reward-SQL: Boosting Text-to-SQL via Stepwise Reasoning and Process-Supervised Rewards

TL;DR

Reward-SQL introduces a structured Chain-of-CTEs framework for stepwise Text-to-SQL reasoning and systematically studies how to integrate Process-supervised Reward Models (PRMs) into both training and inference. Through a three-stage pipeline—Model Initialization, Online RL Training (GRPO), and PRM-assisted Inference (Best-of-N)—the approach achieves strong execution accuracy on the BIRD benchmark, notably 68.9% with a 7B model, and generalizes to Spider in zero-shot settings. Key findings show online GRPO plus PRM-guided inference yields the best performance, offline DPO can outperform rejection sampling, and test-time Best-of-N significantly boosts output quality. The work demonstrates that careful design of stepwise rewards and structured reasoning can meaningfully enhance Text-to-SQL systems, with practical implications for scalable, interpretable SQL generation.

Abstract

Recent advances in large language models (LLMs) have significantly improved performance on the Text-to-SQL task by leveraging their powerful reasoning capabilities. To enhance accuracy during the reasoning process, external Process Reward Models (PRMs) can be introduced during training and inference to provide fine-grained supervision. However, if misused, PRMs may distort the reasoning trajectory and lead to suboptimal or incorrect SQL generation. To address this challenge, we propose Reward-SQL, a framework that systematically explores how to incorporate PRMs into the Text-to-SQL reasoning process effectively. Our approach follows a "cold start, then PRM supervision" paradigm. Specifically, we first train the model to decompose SQL queries into structured stepwise reasoning chains using common table expressions (Chain-of-CTEs), establishing a strong and interpretable reasoning baseline. Then, we investigate four strategies for integrating PRMs, and find that combining PRM as an online training signal (e.g.,GRPO) with PRM-guided inference (e.g., best-of-N sampling) yields the best results. Empirically, on the BIRD benchmark, Reward-SQL enables models supervised by PRM (7B) to achieve a 13.1% performance gain across various guidance strategies. Notably, our GRPO-aligned policy model based on Qwen2.5-Coder-7B-Instruct achieves 68.9% accuracy on the BIRD development set, outperforming all baseline methods under the same model size. These results demonstrate the effectiveness of Reward-SQL in leveraging reward-based supervision for Text-to-SQL reasoning.

Figures (11)

• Figure 1: Stepwise SQL Reasoning via Chain-of-CTEs. Unlike direct SQL generation, our proposed Chain-of-CTEs approach decomposes Text-to-SQL into step-by-step reasoning using Common Table Expressions (CTEs), and leverages Process-supervised Reward Models (PRMs) to provide fine-grained evaluation of each intermediate step.
• Figure 2: Overview of the Reward-SQL framework. Our approach comprises three key stages: 1) Model Initialization, where the model learns to decompose SQL generation into a structured Chain-of-CTEs (Section \ref{['sec:04model init']}); 2) Online RL Training, where the PRM guides the policy model through online reinforcement learning via GRPO (Section \ref{['sec:05 online training']}); and 3) PRM-Assisted Inference, where we leverage the PRM to enhance inference quality by test-time scaling (Section \ref{['sec:05 inference']}).
• Figure 3: Optimization Patterns in the PR-OR Space Across Different Approaches.
• Figure 4: Performance fluctuation during GRPO training with DeepSeekMath's reward formulation. The model exhibits high volatility without consistent improvement on challenging test sets, indicating optimization instability due to reward hacking.
• Figure 5: Example of reward hacking behavior across training epochs. As training progresses, the model generates increasingly shorter solutions to maximize rewards, sacrificing solution quality and correctness. The figure shows how the same query receives progressively simplified responses that ultimately fail to solve the problem correctly.

Theorems & Definitions (2)

• Definition 1: Common Table Expression
• Definition 2: Chain-of-CTEs