Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SecureCodeRL: Security-Aware Reinforcement Learning for Code Generation with Partial-Credit Rewards

Suryansh Singh Sijwali, Suman Saha

TL;DR

The paper addresses the challenge of generating correct code with large language models under strict stdin/stdout judging while also considering security. It introduces SecureCodeRL, a reinforcement learning pipeline that optimizes a combined reward $R = \alpha R_{\text{func}} + \beta R_{\text{sec}}$, where a partial-credit functional reward mitigates reward sparsity and a Bandit-based security penalty discourages insecure patterns. Through benchmarking on APPS+ and a pilot RL study, the approach demonstrates improved syntax validity and a non-zero test-signal (albeit small) while maintaining Bandit-clean results, suggesting reward shaping can enable learning where binary signals fail. The work provides a practical path toward more reliable and secure code generation, while acknowledging the need for larger-scale evaluation, uncertainty quantification, and broader security analysis to validate and extend these findings.

Abstract

Large Language Models (LLMs) can generate plausible code, but in settings that require exact stdin/stdout behavior they frequently produce programs that compile yet fail tests, and in some cases they introduce security-sensitive patterns. This paper presents SecureCodeRL, a reinforcement learning (RL) pipeline for security-aware code generation that optimizes a combined reward R = αRfunc + \b{eta}Rsec. The key idea is a partial-credit functional reward that assigns intermediate scores for syntactic validity, successful execution, and producing output, reducing reward sparsity that otherwise stalls learning on competitive programming style tasks. I evaluate supervised fine-tuning (SFT) and PPO variants on a small held-out prompt set from APPS+ and observe that PPO with partial credit (using a continued-training variant) improves syntax validity from 45% (SFT) to 60% and achieves the only non-zero test success signal in this pilot evaluation (5% at-least-one-test-pass), while remaining 100% clean under Bandit static analysis. Although Bandit findings were absent in this small evaluation, the security term is integrated into training to discourage insecure shortcuts when they appear.

SecureCodeRL: Security-Aware Reinforcement Learning for Code Generation with Partial-Credit Rewards

TL;DR

The paper addresses the challenge of generating correct code with large language models under strict stdin/stdout judging while also considering security. It introduces SecureCodeRL, a reinforcement learning pipeline that optimizes a combined reward , where a partial-credit functional reward mitigates reward sparsity and a Bandit-based security penalty discourages insecure patterns. Through benchmarking on APPS+ and a pilot RL study, the approach demonstrates improved syntax validity and a non-zero test-signal (albeit small) while maintaining Bandit-clean results, suggesting reward shaping can enable learning where binary signals fail. The work provides a practical path toward more reliable and secure code generation, while acknowledging the need for larger-scale evaluation, uncertainty quantification, and broader security analysis to validate and extend these findings.

Abstract

Large Language Models (LLMs) can generate plausible code, but in settings that require exact stdin/stdout behavior they frequently produce programs that compile yet fail tests, and in some cases they introduce security-sensitive patterns. This paper presents SecureCodeRL, a reinforcement learning (RL) pipeline for security-aware code generation that optimizes a combined reward R = αRfunc + \b{eta}Rsec. The key idea is a partial-credit functional reward that assigns intermediate scores for syntactic validity, successful execution, and producing output, reducing reward sparsity that otherwise stalls learning on competitive programming style tasks. I evaluate supervised fine-tuning (SFT) and PPO variants on a small held-out prompt set from APPS+ and observe that PPO with partial credit (using a continued-training variant) improves syntax validity from 45% (SFT) to 60% and achieves the only non-zero test success signal in this pilot evaluation (5% at-least-one-test-pass), while remaining 100% clean under Bandit static analysis. Although Bandit findings were absent in this small evaluation, the security term is integrated into training to discourage insecure shortcuts when they appear.
Paper Structure (28 sections, 2 equations, 2 figures, 4 tables)

This paper contains 28 sections, 2 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Contributions
  3. Background and Related Work
  4. Code generation and execution feedback
  5. Security analysis as a training signal
  6. Dataset, Benchmark, and Failure Modes
  7. Multi-model benchmark: the gap between "valid" and "correct"
  8. Failure taxonomy: what actually goes wrong
  9. Qualitative failure modes (representative examples)
  10. Failure mode A: the "missing print" bug (silent programs)
  11. Failure mode B: wrong-format output
  12. Failure mode C: insecure shortcuts (e.g., eval)
  13. Method
  14. Problem formulation
  15. Security reward: $R_{\text{sec}}$
  16. ...and 13 more sections

Figures (2)

  • Figure 1: SecureCodeRL pipeline: SFT (LoRA) initializes a policy, then PPO optimizes a combined reward. Functional and security rewards merge into a single signal before updating the PPO policy.
  • Figure 2: Partial-credit functional reward: instead of binary test pass/fail, I assign intermediate scores for syntax validity, successful execution, and producing output, with full credit reserved for passing tests.