Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CVeDRL: An Efficient Code Verifier via Difficulty-aware Reinforcement Learning

Ji Shi, Peiming Guo, Meishan Zhang, Miao Zhang, Xuebo Liu, Min Zhang, Weili Guan

TL;DR

CVeDRL addresses the challenge of building efficient code verifiers for LLM-generated code by moving beyond purely supervised training to reinforcement learning guided by syntax and functionality rewards. The method introduces branch- and sample-difficulty awareness through exponential reward shaping and static code metrics, underpinned by a theoretical analysis that links test pass rates, branch coverage, and verification confidence. Empirical results show CVeDRL-0.6B achieves state-of-the-art verification performance across multiple benchmarks with substantial inference speedups, while producing high-quality unit tests with fewer redundant assertions. The work demonstrates practical impact by enabling faster, more reliable code verification with compact models, though it notes limitations related to partial-code scenarios and broader language support.

Abstract

Code verifiers play a critical role in post-verification for LLM-based code generation, yet existing supervised fine-tuning methods suffer from data scarcity, high failure rates, and poor inference efficiency. While reinforcement learning (RL) offers a promising alternative by optimizing models through execution-driven rewards without labeled supervision, our preliminary results show that naive RL with only functionality rewards fails to generate effective unit tests for difficult branches and samples. We first theoretically analyze showing that branch coverage, sample difficulty, syntactic and functional correctness can be jointly modeled as RL rewards, where optimizing these signals can improve the reliability of unit-test-based verification. Guided by this analysis, we design syntax- and functionality-aware rewards and further propose branch- and sample-difficulty--aware RL using exponential reward shaping and static analysis metrics. With this formulation, CVeDRL achieves state-of-the-art performance with only 0.6B parameters, yielding up to 28.97% higher pass rate and 15.08% higher branch coverage than GPT-3.5, while delivering over $20\times$ faster inference than competitive baselines. Code is available at https://github.com/LIGHTCHASER1/CVeDRL.git

CVeDRL: An Efficient Code Verifier via Difficulty-aware Reinforcement Learning

TL;DR

CVeDRL addresses the challenge of building efficient code verifiers for LLM-generated code by moving beyond purely supervised training to reinforcement learning guided by syntax and functionality rewards. The method introduces branch- and sample-difficulty awareness through exponential reward shaping and static code metrics, underpinned by a theoretical analysis that links test pass rates, branch coverage, and verification confidence. Empirical results show CVeDRL-0.6B achieves state-of-the-art verification performance across multiple benchmarks with substantial inference speedups, while producing high-quality unit tests with fewer redundant assertions. The work demonstrates practical impact by enabling faster, more reliable code verification with compact models, though it notes limitations related to partial-code scenarios and broader language support.

Abstract

Code verifiers play a critical role in post-verification for LLM-based code generation, yet existing supervised fine-tuning methods suffer from data scarcity, high failure rates, and poor inference efficiency. While reinforcement learning (RL) offers a promising alternative by optimizing models through execution-driven rewards without labeled supervision, our preliminary results show that naive RL with only functionality rewards fails to generate effective unit tests for difficult branches and samples. We first theoretically analyze showing that branch coverage, sample difficulty, syntactic and functional correctness can be jointly modeled as RL rewards, where optimizing these signals can improve the reliability of unit-test-based verification. Guided by this analysis, we design syntax- and functionality-aware rewards and further propose branch- and sample-difficulty--aware RL using exponential reward shaping and static analysis metrics. With this formulation, CVeDRL achieves state-of-the-art performance with only 0.6B parameters, yielding up to 28.97% higher pass rate and 15.08% higher branch coverage than GPT-3.5, while delivering over faster inference than competitive baselines. Code is available at https://github.com/LIGHTCHASER1/CVeDRL.git
Paper Structure (47 sections, 36 equations, 3 figures, 7 tables)

This paper contains 47 sections, 36 equations, 3 figures, 7 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Enhancement of code generation via unit testing.
  4. Unit test generation.
  5. Method
  6. Unit Test Majority-Voting Framework and RL Insights
  7. Base Code Verifier by RL
  8. Syntax Reward.
  9. Functionality Reward.
  10. Branch-Difficulty-aware RL
  11. Sample-Difficulty-aware RL
  12. Experiments
  13. Experimental Setup
  14. Datasets.
  15. Metrics.
  16. ...and 32 more sections

Figures (3)

  • Figure 1: (a) We design syntax and functionality rewards and employ GRPO to train the base code verifier. Base model struggles to produce effective unit test cases for the difficult branches and samples. Therefore, we propose Branch-Difficulty-aware and Sample-Difficulty-aware reinforcement learning based on exponential reward shaping and static analysis metrics (Halstead Complexity and Maintainability Index). (b) Exponential reward shaping modifies the coverage reward function from a linear format into an exponential format.
  • Figure 2: Performance of three unit‐test generators at different test‐case scales, with LLaMA3-8B on HumanEval+ and GPT-4o-mini on MBPP+ as policy model separately.
  • Figure 3: A case of the training pipeline for CVeDRL.