Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifiers

TL;DR

This work addresses the limitation of value-free RL methods by reintegrating a verification signal through a unified generative verifier trained alongside the RL reasoner. The proposed RL$^V$ framework jointly optimizes the RL objective and a verification objective within a single LLM, enabling test-time verification without the overhead of separate verifiers or value networks. Empirically, RL$^V$ yields over a $20\%$ improvement in MATH accuracy with parallel sampling and enables $8$–$32\times$ faster test-time scaling, with strong generalization to harder problems and out-of-domain tasks, and extends benefits to long CoT models. The approach shows synergy between verification and RL objectives, enabling flexible test-time strategies (weighted voting, BoN) and dynamic allocation of sequential compute, thereby offering a practical and scalable path for enhancing reasoning in LLMs.

Abstract

Prevalent reinforcement learning~(RL) methods for fine-tuning LLM reasoners, such as GRPO or Leave-one-out PPO, abandon the learned value function in favor of empirically estimated returns. This hinders test-time compute scaling that relies on using the value-function for verification. In this work, we propose RL$^V$ that augments any ``value-free'' RL method by jointly training the LLM as both a reasoner and a generative verifier using RL-generated data, adding verification capabilities without significant overhead. Empirically, RL$^V$ boosts MATH accuracy by over 20\% with parallel sampling and enables $8-32\times$ efficient test-time compute scaling compared to the base RL method. RL$^V$ also exhibits strong generalization capabilities for both easy-to-hard and out-of-domain tasks. Furthermore, RL$^V$ achieves $1.2-1.6\times$ higher performance when jointly scaling parallel and sequential test-time compute with a long reasoning R1 model.

Figures (9)

• Figure 1: Left: Scaling Sequential and Parallel Compute Jointly with GRPO$^V$ compared to baselines on the AIME'24 using R1-Distill-Qwen-1.5B as the base LLM. We use Hendrycks' MATH for RL fine-tuning. Each point represents the compute-optimal accuracy achieved at a sequence length using 64 parallel samples. Right: Length Selection Using a Joint Verifier. By iteratively increasing the generation length until a chosen RL$^V$ confidence threshold is met, we can obtain the maximum accuracy at a given sequential compute budget, allowing the model to dynamically allocate more sequential compute to difficult problems.
• Figure 2: RL$^V$ offers significant compute efficiency and performance gains over base "value-free" RL methods when scaling test-time compute with weighted majority voting on MATH500 lightman2023let. For scoring solutions, we use LLM-as-a-Judge as the verifier for the base method, while the trained unified verifier for RL$^V$. These results are based on RL fine-tuning Qwen2.5-Math-1.5B on Hendrycks MATH.
• Figure 3: Overview of RL$^V$: (Top) During training, the LLM policy generates solutions $y$. This data is used for policy updates with RL and simultaneously trains the same LLM as a generative verifier via supervised fine-tuning (SFT) on correctness labels by asking the model 'Is this solution correct? Answer Yes or No'. (Bottom) At test time, the unified LLM generates N solutions and also acts as a verifier to assign scores for re-ranking using Best-of-N or weighted voting.
• Figure 4: RL$^V$ outperforms the base RL method (Leave-One-Out-PPO) consistently across different number of solutions for different generalization settings with respect to the MATH training dataset. (Left) In-distribution Generalization on MATH500. (Center). Easy-to-Hard Generalization on MATH$^2$(Right). Out-of-Domain Generalization on Physics problems in the GPQA Diamond split.
• Figure 5: Comparing Test-Time Compute Strategies Evaluating verifier-based answer selection strategies (weighted voting, Best-of-N) and verifier-free majority voting on AIME'24. The optimal strategy differs for the short CoT and long CoT tuned models.