Secure Code Generation via Online Reinforcement Learning with Vulnerability Reward Model

TL;DR

The paper targets insecure code produced by large language models and proposes SecCoderX, an online reinforcement learning framework that preserves code functionality while enhancing security. It integrates two key innovations: reality-grounded vulnerability-inducing task synthesis to create realistic RL prompts, and a CWE-conditioned vulnerability reward model that provides scalable security supervision. These components are fused in an online RL loop with a composite reward that couples security improvements with maintenance of pre-alignment functionality, enabling end-to-end secure code generation. Empirical results show SecCoderX achieving higher Effective Safety Rate than unaligned models and surpassing prior methods that degrade functional performance, along with releasing a substantial vulnerability-inducing prompt dataset and an 8B vulnerability reward model. The work demonstrates practical potential for deploying secure AI-assisted software development without sacrificing usability, and it contributes a scalable framework for future security-aligned code generation research.

Abstract

Large language models (LLMs) are increasingly used in software development, yet their tendency to generate insecure code remains a major barrier to real-world deployment. Existing secure code alignment methods often suffer from a functionality--security paradox, improving security at the cost of substantial utility degradation. We propose SecCoderX, an online reinforcement learning framework for functionality-preserving secure code generation. SecCoderX first bridges vulnerability detection and secure code generation by repurposing mature detection resources in two ways: (i) synthesizing diverse, reality-grounded vulnerability-inducing coding tasks for online RL rollouts, and (ii) training a reasoning-based vulnerability reward model that provides scalable and reliable security supervision. Together, these components are unified in an online RL loop to align code LLMs to generate secure and functional code. Extensive experiments demonstrate that SecCoderX achieves state-of-the-art performance, improving Effective Safety Rate (ESR) by approximately 10% over unaligned models, whereas prior methods often degrade ESR by 14-54%. We release our code, dataset and model checkpoints at https://github.com/AndrewWTY/SecCoderX.

Figures (19)

• Figure 1: Average Effective Safety Rate (ESR) of secure code alignment methods across multiple models on secure code generation benchmarks. ESR is a composite metric quantifying secure utility via weighting the safety rate by its functional correctness.
• Figure 2: Overview of the SecCoderX Training Pipeline. The framework proceeds in three stages: (A) Reality-Grounded Vulnerability-Inducing Coding Task Synthesis: We repurpose vulnerability detection datasets by first querying a strong LLM to infer N plausible repositories where each vulnerable code snippet's logic could naturally arise. We then synthesize vulnerability-inducing coding prompts conditioned on these repositories with a randomly selected programming language to create a dataset for Online RL alignment. (B) Vulnerability Reward Model Training: We augment the vulnerability detection data by distilling structural vulnerability detection reasoning from a teacher model, which is used to train a reasoning-based, CWE-conditioned vulnerability reward model. (C) Online RL Alignment: Leveraging the synthesized prompts from (A) and the reward signal from (B), we align the target model via online reinforcement learning with a specially designed reward system to generate code that is both secure and functionally correct.
• Figure 3: Comparison of Execution Time vs. F1 score on multiple vulnerability detection benchmarks (PrimeVul, SVEN, ProSec, and R2Vul) between SAST tools and SecCoderX RM across different (base and high) hardware configurations.
• Figure 4: Code Vulnerable to CWE-787: Out-of-bounds Write
• Figure 5: Original Qwen3-8B's Vulnerability Detection Analysis