Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignment

Xue Jiang, Yihong Dong, Mengyang Liu, Hongyi Deng, Tian Wang, Yongding Tao, Rongyu Cao, Binhua Li, Zhi Jin, Wenpin Jiao, Fei Huang, Yongbin Li, Ge Li

TL;DR

CodeRL+ tackles the fundamental gap between textual code generation and correct execution by injecting execution semantics alignment into reinforcement learning with verifiable rewards. It jointly optimizes code generation and inference of execution trajectories, using failed rollouts to derive dense alignment prompts and a mixed objective that rewards both functional correctness and semantic understanding. Across multiple benchmarks (HumanEval, LeetCode, LiveCodeBench) and model families, CodeRL+ achieves state-of-the-art results and demonstrates strong generalization to code reasoning and test-output generation tasks, with extensive ablations and probing showing improved internal alignment between text and runtime semantics. The approach offers a practical, data-efficient method to enhance both code correctness and reasoning in LLM-based code generation, compatible with diverse RL algorithms and model scales.

Abstract

While Large Language Models (LLMs) excel at code generation by learning from vast code corpora, a fundamental semantic gap remains between their training on textual patterns and the goal of functional correctness, which is governed by formal execution semantics. Reinforcement Learning with Verifiable Rewards (RLVR) approaches attempt to bridge this gap using outcome rewards from executing test cases. However, solely relying on binary pass/fail signals is inefficient for establishing a well-aligned connection between the textual representation of code and its execution semantics, especially for subtle logical errors within the code. In this paper, we propose CodeRL+, a novel approach that integrates execution semantics alignment into the RLVR training pipeline for code generation. CodeRL+ enables the model to infer variable-level execution trajectory, providing a direct learning signal of execution semantics. CodeRL+ can construct execution semantics alignment directly using existing on-policy rollouts and integrates seamlessly with various RL algorithms. Extensive experiments demonstrate that CodeRL+ outperforms post-training baselines (including RLVR and Distillation), achieving a 4.6% average relative improvement in pass@1. CodeRL+ generalizes effectively to other coding tasks, yielding 15.5% and 4.4% higher accuracy on code-reasoning and test-output-generation benchmarks, respectively. CodeRL+ shows strong applicability across diverse RL algorithms and LLMs. Furthermore, probe analyses provide compelling evidence that CodeRL+ strengthens the alignment between code's textual representations and its underlying execution semantics.

CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignment

TL;DR

CodeRL+ tackles the fundamental gap between textual code generation and correct execution by injecting execution semantics alignment into reinforcement learning with verifiable rewards. It jointly optimizes code generation and inference of execution trajectories, using failed rollouts to derive dense alignment prompts and a mixed objective that rewards both functional correctness and semantic understanding. Across multiple benchmarks (HumanEval, LeetCode, LiveCodeBench) and model families, CodeRL+ achieves state-of-the-art results and demonstrates strong generalization to code reasoning and test-output generation tasks, with extensive ablations and probing showing improved internal alignment between text and runtime semantics. The approach offers a practical, data-efficient method to enhance both code correctness and reasoning in LLM-based code generation, compatible with diverse RL algorithms and model scales.

Abstract

While Large Language Models (LLMs) excel at code generation by learning from vast code corpora, a fundamental semantic gap remains between their training on textual patterns and the goal of functional correctness, which is governed by formal execution semantics. Reinforcement Learning with Verifiable Rewards (RLVR) approaches attempt to bridge this gap using outcome rewards from executing test cases. However, solely relying on binary pass/fail signals is inefficient for establishing a well-aligned connection between the textual representation of code and its execution semantics, especially for subtle logical errors within the code. In this paper, we propose CodeRL+, a novel approach that integrates execution semantics alignment into the RLVR training pipeline for code generation. CodeRL+ enables the model to infer variable-level execution trajectory, providing a direct learning signal of execution semantics. CodeRL+ can construct execution semantics alignment directly using existing on-policy rollouts and integrates seamlessly with various RL algorithms. Extensive experiments demonstrate that CodeRL+ outperforms post-training baselines (including RLVR and Distillation), achieving a 4.6% average relative improvement in pass@1. CodeRL+ generalizes effectively to other coding tasks, yielding 15.5% and 4.4% higher accuracy on code-reasoning and test-output-generation benchmarks, respectively. CodeRL+ shows strong applicability across diverse RL algorithms and LLMs. Furthermore, probe analyses provide compelling evidence that CodeRL+ strengthens the alignment between code's textual representations and its underlying execution semantics.
Paper Structure (27 sections, 6 equations, 6 figures, 3 tables)

This paper contains 27 sections, 6 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Reinforcement Learning for Code Generation
  4. Learning Program Executions with Large Language Models
  5. Methodology
  6. Preliminaries and Definition
  7. Policy Gradient Optimization.
  8. Execution Semantics
  9. CodeRL+
  10. Experiments
  11. Experiment Setup
  12. Training Details.
  13. Evaluation Details.
  14. Baselines.
  15. Experiment Results
  16. ...and 12 more sections

Figures (6)

  • Figure 1: Illustrations of the existing RLVR struggling to establish a well-aligned connection between the textual representation of code and its execution semantics. (a) An example of execution trace inference. (b) Result of execution trace inference task.
  • Figure 2: Overall Pipeline of of CodeRL+.
  • Figure 3: Result of Ablation Study.
  • Figure 4: Training dynamics of CodeRL+ and GRPO.
  • Figure 5: Probing results of CodeRL+ on HumanEval code generation benchmark.
  • ...and 1 more figures