Secure Code Generation at Scale with Reflexion

TL;DR

The paper tackles insecure code generation by large language models and evaluates a scalable, CWE-aligned approach using the corrected Instruct Prime benchmark. It introduces a three-round reflexion prompting strategy guided by Insecure Code Detector feedback to revise code across eight languages and five code LLMs. Key findings show that initial insecurity rates are roughly 25–33%, with reflexion boosting secure-generation accuracy from about 70.7% to 79.4%, most gains occurring after the first round. The work provides a replication package, analyzes cross-language and CWE-type patterns, and offers practical guidance for deploying reflexion-based security improvements in real-world code generation workflows.

Abstract

Large language models (LLMs) are now widely used to draft and refactor code, but code that works is not necessarily secure. We evaluate secure code generation using the Instruct Prime, which eliminated compliance-required prompts and cue contamination, and evaluate five instruction-tuned code LLMs using a zero-shot baseline and a three-round reflexion prompting approach. Security is measured using the Insecure Code Detector (ICD), and results are reported by measuring Repair, Regression, and NetGain metrics, considering the programming language and CWE family. Our findings show that insecurity remains common at the first round: roughly 25-33% of programs are insecure at a zero-shot baseline (t0 ). Weak cryptography/config-dependent bugs are the hardest to avoid while templated ones like XSS, code injection, and hard-coded secrets are handled more reliably. Python yields the highest secure rates; C and C# are the lowest, with Java, JS, PHP, and C++ in the middle. Reflexion prompting improves security for all models, improving average accuracy from 70.74% at t0 to 79.43% at t3 , with the largest gains in the first round followed by diminishing returns. The trends with Repair, Regression, and NetGain metrics show that applying one to two rounds produces most of the benefits. A replication package is available at https://doi.org/10.5281/zenodo.17065846.

Figures (3)

• Figure 1: Overview of our evaluation workflow. Prompts $P_0$ from the Instruct Prime dataset are selected (Step 1) and sent to a code LLM (Step 2) to generate outputs $t_i$ (Step 3), which are checked by the Insecure Code Detector (ICD) (Step 4). In the zero-shot path, the pipeline terminates with an ICD label for $t_0$ (Step 6). In the reflexion path, the code and ICD feedback are folded into a revised prompt $P_{i+1}$ (Step 5), and Steps 2--5 repeat for three rounds to yield $t_1$, $t_2$, and $t_3$. Finally, in Step 7, we aggregate and analyze these results to answer RQ1/RQ2.
• Figure 2: Secure Code Generation Performance Range Across Models by CWE Type (Top 30 by Sample Count)
• Figure 3: Three rounds of reflexion ($t_1$ -- $t_3$) vs the $t_0$ baseline across five code LLMs. Lines show fixes (Repair), new vulnerabilities (Regression), and net gain (Repair - Regression). Points mark accuracy at each step $t_i$.