VeRPO: Verifiable Dense Reward Policy Optimization for Code Generation

TL;DR

VeRPO tackles the sparse reward problem in RL for code generation by deriving dense, verifiable rewards from weighted partial success, where unit-test difficulties are estimated online and normalized to prevent overemphasis on easy tests. The approach anchors dense turn-level signals with a global end-to-end execution outcome and integrates them through a dual-level advantage framework within a group-based RL setting. Empirical results across multiple benchmarks show VeRPO achieving up to 8.83 percentage points gain in pass@1 with negligible time cost and zero GPU memory overhead, outperforming both outcome-driven baselines and RM-based dense rewards, while maintaining stability in multi-turn optimization. The work provides a scalable, model-free alternative that leverages verifiable execution feedback to deliver robust learning signals for code-generation policies, with broad implications for improving RL-based code assistants.

Abstract

Effective reward design is a central challenge in Reinforcement Learning (RL) for code generation. Mainstream pass/fail outcome rewards enforce functional correctness via executing unit tests, but the resulting sparsity limits potential performance gains. While recent work has explored external Reward Models (RM) to generate richer, continuous rewards, the learned RMs suffer from reward misalignment and prohibitive computational cost. In this paper, we introduce \textbf{VeRPO} (\textbf{V}erifiable D\textbf{e}nse \textbf{R}eward \textbf{P}olicy \textbf{O}ptimization), a novel RL framework for code generation that synthesizes \textit{robust and dense rewards fully grounded in verifiable execution feedback}. The core idea of VeRPO is constructing dense rewards from weighted partial success: by dynamically estimating the difficulty weight of each unit test based on the execution statistics during training, a dense reward is derived from the sum of weights of the passed unit tests. To solidify the consistency between partial success and end-to-end functional correctness, VeRPO further integrates the dense signal with global execution outcomes, establishing a robust and dense reward paradigm relying solely on verifiable execution feedback. Extensive experiments across diverse benchmarks and settings demonstrate that VeRPO consistently outperforms outcome-driven and RM-based baselines, achieving up to +8.83\% gain in pass@1 with negligible time cost (< 0.02\%) and zero GPU memory overhead.

Figures (6)

• Figure 1: Comparison of reward design in RL for code generation.
• Figure 2: Overview of VeRPO. VeRPO fuses turn-level dense rewards derived from dynamically weighted partial success with global trajectory-level outcomes, enabling robust and dense policy optimization for code generation.
• Figure 3: Signal efficiency analysis of GRPO and VeRPO. The degenerate group ratio represents the fraction of groups yielding zero advantage within a training batch. Lower values indicate more efficient signal utilization.
• Figure 4: Computational analysis of VeRPO. Components shared with GRPO are highlighted in blue, while VeRPO-specific computations are shown in orange. A broken y-axis scale is employed to enhance the visualization of small values.
• Figure 5: The visualization of the average ground-truth pass rate computed on the online training rollout batches.