ConstrainedSQL: Training LLMs for Text2SQL via Constrained Reinforcement Learning

TL;DR

ConstrainedSQL tackles reward hacking in Text2SQL RL by formulating training as constrained RL with natural reward and constraint signals and a KL penalty against a reference policy. It provides a theoretical guarantee on the primal-dual gap via a parameterization gap ν, expressed as $0 \le D^* - P^* \le (\beta + B + B \|\lambda_\nu^*\|_1) \nu$. Empirically, it outperforms state-of-the-art RL-based Text2SQL methods on Spider and BIRD benchmarks and matches SFT-based performance while requiring far fewer RL samples. The work demonstrates improved reliability and interpretability of post-training NL-to-SQL reasoning in LLMs.

Abstract

Reinforcement learning (RL) has demonstrated significant promise in enhancing the reasoning capabilities of Text2SQL LLMs, especially with advanced algorithms such as GRPO and DAPO. However, the performance of these methods is highly sensitive to the design of reward functions. Inappropriate rewards can lead to reward hacking, where models exploit loopholes in the reward structure to achieve high scores without genuinely solving the task. This work considers a constrained RL framework for Text2SQL that incorporates natural and interpretable reward and constraint signals, while dynamically balancing trade-offs among them during the training. We establish the theoretical guarantees of our constrained RL framework and our numerical experiments on the well-known Text2SQL datasets substantiate the improvement of our approach over the state-of-the-art RL-trained LLMs.

Figures (3)

• Figure 1: Training curves of the reward and constraints in ConstrainedSQL.
• Figure 2: The example of the prompt in ConstrainedSQL.
• Figure 3: The example of the response in ConstrainedSQL.

Theorems & Definitions (2)

• Theorem 1: Primal-dual gap
• Lemma 1