Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

How Secure is Secure Code Generation? Adversarial Prompts Put LLM Defenses to the Test

Melissa Tessa, Iyiola E. Olatunji, Aicha War, Jacques Klein, Tegawendé F. Bissyandé

TL;DR

The paper tackles the problem that secure code-generation methods often report strong security under separate, non-unified benchmarks. It presents the first adversarial audit of three leading methods (Sven, SafeCoder, PromSec) under realistic prompt perturbations and evaluates security and functionality jointly using CodeSecEval. Static analyzers overestimate security by large factors, and a substantial fraction of outputs labeled secure are non-functional, with true secure-and-functional rates dropping to as low as $3\%$ to $17\%$ under attack. The results motivate best practices, including explicit threat models, adversarial robustness testing, unified joint evaluation, consensus-based validation, robust reporting, calibrated LLM judges, and a shift toward semantic security reasoning. Together, these contributions highlight the need for rigorous, deployment-oriented evaluation standards to ensure that secure code-generation methods deliver trustworthy code in real-world software supply chains.

Abstract

Recent secure code generation methods, using vulnerability-aware fine-tuning, prefix-tuning, and prompt optimization, claim to prevent LLMs from producing insecure code. However, their robustness under adversarial conditions remains untested, and current evaluations decouple security from functionality, potentially inflating reported gains. We present the first systematic adversarial audit of state-of-the-art secure code generation methods (SVEN, SafeCoder, PromSec). We subject them to realistic prompt perturbations such as paraphrasing, cue inversion, and context manipulation that developers might inadvertently introduce or adversaries deliberately exploit. To enable fair comparison, we evaluate all methods under consistent conditions, jointly assessing security and functionality using multiple analyzers and executable tests. Our findings reveal critical robustness gaps: static analyzers overestimate security by 7 to 21 times, with 37 to 60% of ``secure'' outputs being non-functional. Under adversarial conditions, true secure-and-functional rates collapse to 3 to 17%. Based on these findings, we propose best practices for building and evaluating robust secure code generation methods. Our code is available.

How Secure is Secure Code Generation? Adversarial Prompts Put LLM Defenses to the Test

TL;DR

The paper tackles the problem that secure code-generation methods often report strong security under separate, non-unified benchmarks. It presents the first adversarial audit of three leading methods (Sven, SafeCoder, PromSec) under realistic prompt perturbations and evaluates security and functionality jointly using CodeSecEval. Static analyzers overestimate security by large factors, and a substantial fraction of outputs labeled secure are non-functional, with true secure-and-functional rates dropping to as low as to under attack. The results motivate best practices, including explicit threat models, adversarial robustness testing, unified joint evaluation, consensus-based validation, robust reporting, calibrated LLM judges, and a shift toward semantic security reasoning. Together, these contributions highlight the need for rigorous, deployment-oriented evaluation standards to ensure that secure code-generation methods deliver trustworthy code in real-world software supply chains.

Abstract

Recent secure code generation methods, using vulnerability-aware fine-tuning, prefix-tuning, and prompt optimization, claim to prevent LLMs from producing insecure code. However, their robustness under adversarial conditions remains untested, and current evaluations decouple security from functionality, potentially inflating reported gains. We present the first systematic adversarial audit of state-of-the-art secure code generation methods (SVEN, SafeCoder, PromSec). We subject them to realistic prompt perturbations such as paraphrasing, cue inversion, and context manipulation that developers might inadvertently introduce or adversaries deliberately exploit. To enable fair comparison, we evaluate all methods under consistent conditions, jointly assessing security and functionality using multiple analyzers and executable tests. Our findings reveal critical robustness gaps: static analyzers overestimate security by 7 to 21 times, with 37 to 60% of ``secure'' outputs being non-functional. Under adversarial conditions, true secure-and-functional rates collapse to 3 to 17%. Based on these findings, we propose best practices for building and evaluating robust secure code generation methods. Our code is available.
Paper Structure (44 sections, 3 figures, 10 tables)

This paper contains 44 sections, 3 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Secure Code Generation and Benchmarks
  4. Adversarial Attacks on Code LLMs
  5. Threat Model and Scope
  6. Motivation
  7. Threat Model
  8. Research Questions
  9. Auditing Framework
  10. Secure Code Generation Methods Under Audit
  11. Attack Vectors
  12. General-purpose prompt-based attacks
  13. Task-specific attacks
  14. Evaluation Metrics and Datasets
  15. Phase 1: Method-specific metrics.
  16. ...and 29 more sections

Figures (3)

  • Figure 1: Secure code generation audit pipeline.
  • Figure 2: Robustness of SVENsec (secure prefix). Secure code generation rate (%) under adversarial prompt attacks, with relative difference to the unattacked baseline (Original- sec) shown in parentheses.
  • Figure 3: Attack performance on $\text{SVENvul}$ (insecure prefix). Secure code generation rate in % (here, lower is better) under adversarial prompt attacks, with relative difference to the unattacked baseline (Original-vul) shown in parentheses.