Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards

TL;DR

RISE introduces an online reinforcement learning framework that jointly optimizes problem-solving and self-verification for LLMs by leveraging verifiable rewards from an outcome verifier. Implemented with PPO, RISE trains on both solution generation and on-the-fly verification, yielding significant gains in reasoning accuracy and verification accuracy across mathematical benchmarks and model scales. The method enhances test-time performance through verification-aware voting and demonstrates that online verification is crucial for robust self-verification. Findings indicate that verification compute and online coupling are key drivers of improved reasoning, with potential applicability beyond math reasoning to diverse domains requiring verifiable reasoning guarantees.

Abstract

Large Language Models (LLMs) show great promise in complex reasoning, with Reinforcement Learning with Verifiable Rewards (RLVR) being a key enhancement strategy. However, a prevalent issue is ``superficial self-reflection'', where models fail to robustly verify their own outputs. We introduce RISE (Reinforcing Reasoning with Self-Verification), a novel online RL framework designed to tackle this. RISE explicitly and simultaneously trains an LLM to improve both its problem-solving and self-verification abilities within a single, integrated RL process. The core mechanism involves leveraging verifiable rewards from an outcome verifier to provide on-the-fly feedback for both solution generation and self-verification tasks. In each iteration, the model generates solutions, then critiques its own on-policy generated solutions, with both trajectories contributing to the policy update. Extensive experiments on diverse mathematical reasoning benchmarks show that RISE consistently improves model's problem-solving accuracy while concurrently fostering strong self-verification skills. Our analyses highlight the advantages of online verification and the benefits of increased verification compute. Additionally, RISE models exhibit more frequent and accurate self-verification behaviors during reasoning. These advantages reinforce RISE as a flexible and effective path towards developing more robust and self-aware reasoners.

Figures (14)

• Figure 1: Illustration of RISE, which consists of two stages: (i) Problem Solution and Verification Generation: problems from the training batch are used to generate chain-of-thought solutions from the model. Problems and model solutions are then formatted as verification prompts to generate verifications of the solutions. (ii) RL Optimization: the original generation data and their verification are mixed as the new batch, and the model is optimized based on the RL objective.
• Figure 2: Test-time scaling performance across different sampling budgets ("k").
• Figure 3: Comparisons between RISE (self-verify) and off-the-shelf verifiers.
• Figure 4: Reasoning and verification reward at train time.
• Figure 5: Impact of verification data ratio.