Aletheia: What Makes RLVR For Code Verifiers Tick?

TL;DR

This work introduces Aletheia, a controlled testbed for evaluating surrogate code-execution verifiers under covariate shifts and across diverse policy models. It systematically ablates three RLVR recipe components—thinking traces, learning from negative samples, and on-policy training—and shows that GRPO generally yields the strongest verifiers, with thinking traces being essential for the largest models, on-policy learning crucial for small models, and negative samples providing broad stability gains. The study demonstrates that, while offline methods can approach GRPO for certain scales, fully online training remains superior, and simply increasing inference compute cannot fully substitute for core training components. The results offer practical guidance for cost-efficient verifier training and suggest that these insights generalize to cross-domain verifier settings beyond code generation.

Abstract

Multi-domain thinking verifiers trained via Reinforcement Learning from Verifiable Rewards (RLVR) are a prominent fixture of the Large Language Model (LLM) post-training pipeline, owing to their ability to robustly rate and rerank model outputs. However, the adoption of such verifiers towards code generation has been comparatively sparse, with execution feedback constituting the dominant signal. Nonetheless, code verifiers remain valuable toward judging model outputs in scenarios where execution feedback is hard to obtain and are a potentially powerful addition to the code generation post-training toolbox. To this end, we create and open-source Aletheia, a controlled testbed that enables execution-grounded evaluation of code verifiers' robustness across disparate policy models and covariate shifts. We examine components of the RLVR-based verifier training recipe widely credited for its success: (1) intermediate thinking traces, (2) learning from negative samples, and (3) on-policy training. While experiments show the optimality of RLVR, we uncover important opportunities to simplify the recipe. Particularly, despite code verification exhibiting positive training- and inference-time scaling, on-policy learning stands out as the key component at small verifier sizes, and thinking-based training emerges as the most important component at larger scales.

Figures (6)

• Figure 1: The dataset creation pipeline. We follow a three-step process by (1) sourcing competition-level programming questions from CodeContests$^+$wang2025codecontests+ and sampling solutions from open-source LLMs, (2) executing the obtained solutions using SandboxFusion to obtain pass rates (PRs), (3) creating lists of 2-5 codes, of which exactly one is fully correct, and (4) dividing the resulting lists into our training and evaluation sets. A detailed description of the pipeline is in \ref{['sec: data-creation']}.
• Figure 2: Inference-time scaling for our thinking ablation. Chain-of-Thought models do not benefit from additional compute, whereas thinking models see a small upward trend across all model sizes.
• Figure 3: Inference-time scaling plots for ablating on-policy learning. Scaling inference-time compute benefits all models, but cannot fully replace the gains from on-policy learning.
• Figure 4: Inference-time scaling plots for ablating negative samples. The gap between RAFT and GRPO reduces with scaling compute, but persists.
• Figure 5: Reward curves over training for RAFT and GRPO, respectively. Training without negative samples becomes increasingly unstable with scale.