Graph-Reward-SQL: Execution-Free Reinforcement Learning for Text-to-SQL via Graph Matching and Stepwise Reward

TL;DR

This work tackles the inefficiency of reward signals in RL-based Text-to-SQL by proposing Graph-Reward-SQL, which replaces execution-based signals with a fast, execution-free GMNScore and augments it with StepRTM, a stepwise reward for CTE subqueries. GMNScore leverages Graph Matching Networks on Relational Operator Tree representations to assess semantic equivalence without querying databases, substantially reducing latency and GPU memory usage. StepRTM provides intermediate supervision by matching ROTs of CTE subqueries to the reference, improving readability and correctness. Across Spider and BIRD benchmarks, the approach yields superior or competitive performance with markedly improved training efficiency and lower resource demands, demonstrating a scalable direction for RL-based text-to-SQL systems.

Abstract

Reinforcement learning (RL) has been widely adopted to enhance the performance of large language models (LLMs) on Text-to-SQL tasks. However, existing methods often rely on execution-based or LLM-based Bradley-Terry reward models. The former suffers from high execution latency caused by repeated database calls, whereas the latter imposes substantial GPU memory overhead, both of which significantly hinder the efficiency and scalability of RL pipelines. To this end, we propose a novel reward model framework for RL-based Text-to-SQL named Graph-Reward-SQL, which employs the GMNScore outcome reward model. We leverage SQL graph representations to provide accurate reward signals while significantly reducing time cost and GPU memory usage. Building on this foundation, we further introduce StepRTM, a stepwise reward model that provides intermediate supervision over Common Table Expression (CTE) subqueries. This encourages both functional correctness and readability of SQL. Extensive comparative and ablation experiments on standard benchmarks, including Spider and BIRD, demonstrate that our method consistently outperforms existing reward models.

Figures (7)

• Figure 1: Graph-Reward-SQL employs PPO, where rewards drive policy updates. An example illustrates a limitation of EX: the generated SQL 'WHERE age > 34' and the reference SQL 'WHERE age >= 34' produce identical results despite their semantic difference. In contrast, our proposed GMNScore leverages graph representation to capture deep semantic similarity. Additionally, a stepwise reward model, StepRTM, that is tailored for CTE SQL addresses the lack of intermediate rewards. A mock stepwise score is provided for illustration; see Figure \ref{['fig:step_relpm_pipeline']} for more details.
• Figure 2: Overview of the StepRTM Stepwise Reward Calculation. (a) The generated SQL $\hat{q}_{\text{cte}}$ is segmented into a sequence of subqueries, with the end index of each subquery recorded. (b) Both the reference SQL query $q^*$ and each subquery are parsed into ROTs (c) A stepwise matching process is performed between the ROTs. At each step, newly matched nodes are identified and used to compute incremental rewards.
• Figure 3: TS Performance of Qwen2.5-Coder-7B/14B-Ins models directly trained by PPO/GRPO.
• Figure 4: Reward score trends on the training set during GRPO training. The green curve corresponds to the execution-based reward model (EX), while the orange curve shows the results using the proposed reward model GMNScore. Both models are trained with the Qwen2.5-Coder-7B-Ins backbone under the GRPO framework. While GMNScore starts from a lower initial score compared to EX, it demonstrates a larger overall increase.
• Figure 5: Actor entropy of GRPO training. Both models are based on Qwen2.5-Coder-7B-Ins. The orange curve corresponds to GMNScore and the green curve to EX. Entropy consistently decreases as the policy becomes more deterministic. GMNScore exhibits a faster reduction in entropy during the early training stages, while both methods converge to similarly low-entropy levels by the end of training.