Reasoning-SQL: Reinforcement Learning with SQL Tailored Partial Rewards for Reasoning-Enhanced Text-to-SQL

TL;DR

This work tackles Text-to-SQL by enabling deep, reasoning-enhanced SQL generation through reinforcement learning. It introduces Reasoning-SQL, a GRPO-based framework that leverages a suite of partial rewards—including schema linking, AI feedback, n-gram similarity, and syntax checks—to overcome reward sparsity and cultivate intrinsic reasoning. Empirical results on BIRD and Spider benchmarks show that smaller open-source models trained with these rewards outperform supervised fine-tuning and compete with or surpass larger proprietary systems while dramatically reducing inference cost. The study also reveals emergent, structured reasoning patterns that outperform hand-crafted chain-of-thought prompts, underscoring the potential of RL-based reasoning for complex, multi-step language-to-database tasks.

Abstract

Text-to-SQL is a challenging task involving multiple reasoning-intensive subtasks, including natural language understanding, database schema comprehension, and precise SQL query formulation. Existing approaches often rely on handcrafted reasoning paths with inductive biases that can limit their overall effectiveness. Motivated by the recent success of reasoning-enhanced models such as DeepSeek R1 and OpenAI o1, which effectively leverage reward-driven self-exploration to enhance reasoning capabilities and generalization, we propose a novel set of partial rewards tailored specifically for the Text-to-SQL task. Our reward set includes schema-linking, AI feedback, n-gram similarity, and syntax check, explicitly designed to address the reward sparsity issue prevalent in reinforcement learning (RL). Leveraging group relative policy optimization (GRPO), our approach explicitly encourages large language models (LLMs) to develop intrinsic reasoning skills necessary for accurate SQL query generation. With models of different sizes, we demonstrate that RL-only training with our proposed rewards consistently achieves higher accuracy and superior generalization compared to supervised fine-tuning (SFT). Remarkably, our RL-trained 14B-parameter model significantly outperforms larger proprietary models, e.g. o3-mini by 4% and Gemini-1.5-Pro-002 by 3% on the BIRD benchmark. These highlight the efficacy of our proposed RL-training framework with partial rewards for enhancing both accuracy and reasoning capabilities in Text-to-SQL tasks.

Figures (9)

• Figure 1: Overview of the GRPO-based Text-to-SQL training pipeline. For each natural language prompt q and its associated database schema, the policy model $\pi_\theta$ generates a group of candidate SQL queries. Each candidate is evaluated using a suite of reward functions to produce a composite reward. These rewards are then used to compute advantages and update the policy via GRPO.
• Figure 2: Example partial reward calculation for a generated SQL query, illustrating how each reward component, execution accuracy, llm-as-a-judge (AI Feedback), syntax check, schema linking, and n-gram similarity, is derived by comparing the candidate query with the gold (ground truth) query.
• Figure 3: An example illustrating the model’s improvement in reasoning. By step 200 of training, the model adopts a structured approach to SQL synthesis and correctly identifies the join condition.
• Figure 4: An example output of our GRPO-trained model.
• Figure 5: An example output of the model during the training process at step 10 and step 200.