RLPR: Extrapolating RLVR to General Domains without Verifiers

TL;DR

RLPR presents a verifier-free reinforcement learning approach that extrapolates RLVR to general-domain reasoning by using the LLM’s own per-token probabilities over reference answers as the reward. It introduces Probability Reward (PR), a debiased, length-robust reward, and an adaptive curriculum via standard deviation filtering to stabilize training. Across seven benchmarks and multiple base models, RLPR yields consistent improvements in both general and mathematical reasoning, outperforming verifier-based and verifier-free baselines in many settings. The work demonstrates that intrinsic model probabilities can serve as high-quality rewards, reducing reliance on domain-specific verifiers and enabling scalable reasoning improvements. It also shows potential for combining PR with existing verifiers in verifiable domains and outlines future directions for multimodal extensions and larger-scale models.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) demonstrates promising potential in advancing the reasoning capabilities of LLMs. However, its success remains largely confined to mathematical and code domains. This primary limitation stems from the heavy reliance on domain-specific verifiers, which results in prohibitive complexity and limited scalability. To address the challenge, our key observation is that LLM's intrinsic probability of generating a correct free-form answer directly indicates its own evaluation of the reasoning reward (i.e., how well the reasoning process leads to the correct answer). Building on this insight, we propose RLPR, a simple verifier-free framework that extrapolates RLVR to broader general domains. RLPR uses the LLM's own token probability scores for reference answers as the reward signal and maximizes the expected reward during training. We find that addressing the high variance of this noisy probability reward is crucial to make it work, and propose prob-to-reward and stabilizing methods to ensure a precise and stable reward from LLM intrinsic probabilities. Comprehensive experiments in four general-domain benchmarks and three mathematical benchmarks show that RLPR consistently improves reasoning capabilities in both areas for Gemma, Llama, and Qwen based models. Notably, RLPR outperforms concurrent VeriFree by 7.6 points on TheoremQA and 7.5 points on Minerva, and even surpasses strong verifier-model-dependent approaches General-Reasoner by 1.6 average points across seven benchmarks.

Figures (7)

• Figure 1: Overall performance on general-domain and mathematical reasoning benchmarks. By simply replacing the rule-based verifier reward of RLVR with the proposed LLM's intrinsic probability reward, RLPR achieves consistent improvements in both mathematical and general domains, even outperforming strong RL methods driven by model-based verifier reward. Average: average accuracy of five benchmarks. Verifier requirements of different methods are listed in parentheses.
• Figure 2: Existing RLVR methods rely on specialized verifiers for each domain, suffering from high complexity and limited scalability. We propose the RLPR framework, which replaces the complex verifier-based reward with a simple probability-based reward generated by the policy model $\pi_\theta$. $Q$: input question, $z$: generated reasoning content before final answer, $y$: generated final answer, $y^*$: reference answer. As shown in the example on the right side, rules and verifier models wrongly label both $y_2$ and $y_3$ as incorrect due to their limited capability of handling natural language complexity.
• Figure 3: Token-level probability visualization, where deeper colors represent higher values. The underlined part highlights that probabilities precisely reflect that response sequence $o2$ incorrectly place option B after A, resulting lower scores at the corresponding positions in the reference answer. The question is shown in Figure \ref{['fig:teaser']}.
• Figure 4: Reward quality comparison. We report the AUC on both math data and general data, and highlight the average score with the dashed line. Qwn: Qwen2.5 models.
• Figure 5: Robustness across different training prompt templates. RLPR yields consistently higher performance compared with VeriFree. Left: average performance on seven benchmarks. Middle: response length. Right: response entropy during training.