JudgeRLVR: Judge First, Generate Second for Efficient Reasoning

TL;DR

The paper addresses inefficiencies in reinforcement learning with verifiable rewards (RLVR), where final-answer correctness often drives verbose, trial-and-error reasoning. It proposes JudgeRLVR, a two-stage judge-then-generate framework that first trains a discriminative judge to assess candidate solutions and then fine-tunes generation initialized from the judge, without explicit length penalties. Empirically, JudgeRLVR improves average accuracy by about +$3.7$–$4.5$ points across in-domain and out-of-domain math and general knowledge benchmarks, while reducing average generation length by roughly $42\%$ on in-domain tasks and improving generalization to diverse domains; ablations show the necessity of the sequential judge-then-generate design and evidence of a shift toward less backtracking in reasoning. The findings suggest that discriminative competence acts as a precursor to efficient generation, enabling LLMs to prune low-value branches early and produce higher-density, more direct reasoning across math, science, coding, and knowledge tasks; this has practical implications for scalable, reliable reasoning in real-world applications.

Abstract

Reinforcement Learning with Verifiable Rewards (RLVR) has become a standard paradigm for reasoning in Large Language Models. However, optimizing solely for final-answer correctness often drives models into aimless, verbose exploration, where they rely on exhaustive trial-and-error tactics rather than structured planning to reach solutions. While heuristic constraints like length penalties can reduce verbosity, they often truncate essential reasoning steps, creating a difficult trade-off between efficiency and verification. In this paper, we argue that discriminative capability is a prerequisite for efficient generation: by learning to distinguish valid solutions, a model can internalize a guidance signal that prunes the search space. We propose JudgeRLVR, a two-stage judge-then-generate paradigm. In the first stage, we train the model to judge solution responses with verifiable answers. In the second stage, we fine-tune the same model with vanilla generating RLVR initialized from the judge. Compared to Vanilla RLVR using the same math-domain training data, JudgeRLVR achieves a better quality--efficiency trade-off for Qwen3-30B-A3B: on in-domain math, it delivers about +3.7 points average accuracy gain with -42\% average generation length; on out-of-domain benchmarks, it delivers about +4.5 points average accuracy improvement, demonstrating enhanced generalization.

Figures (4)

• Figure 1: Qualitative comparison of reasoning patterns between Vanilla RLVR and JudgeRLVR. While outcome-based RL guarantees final answer correctness, it often induces verbose and unstructured exploration (left). JudgeRLVR implicitly regularizes the reasoning process, leading to a direct and coherent solution path (right).
• Figure 2: Pipeline for the two-stage training of JudgeRLVR
• Figure 3: Base SFT perplexity (PPL) evaluated on sampled outputs along training steps. Vanilla RLVR stays stylistically close to Base SFT (flat PPL), while JudgeRLVR (judging stage) exhibits increasing PPL, indicating style transfer.
• Figure 4: Counts(left) and Frequencies(right) of transition/backtracking markers (e.g., but, however, wait) in sampled outputs across JudgeRLVR generating stage training steps.