Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Model to Breach: Towards Actionable LLM-Generated Vulnerabilities Reporting

Cyril Vallez, Alexander Sternfeld, Andrei Kucharavy, Ljiljana Dolamic

TL;DR

This paper assesses the security of code generated by open-weight LLMs used as coding assistants and shows that vulnerabilities persist in realistic scenarios. It introduces Prompt Exposure ($PE$) and Model Exposure ($ME$), CVSS-compatible metrics that quantify the severity and prevalence of vulnerabilities induced by prompts and model choices, respectively. By extending the Asleep at the Keyboard (AATK) benchmark into an automated, CVSS-aware framework and applying static CodeQL analysis, dynamic testing, and human review, it demonstrates that a non-trivial fraction of generated code remains vulnerable across leading models. The work provides a principled method to rank model security via $ME$ and highlights the need for an end-to-end vulnerability reporting and patching pipeline to improve the safety of LLM-generated code in practice.

Abstract

As the role of Large Language Models (LLM)-based coding assistants in software development becomes more critical, so does the role of the bugs they generate in the overall cybersecurity landscape. While a number of LLM code security benchmarks have been proposed alongside approaches to improve the security of generated code, it remains unclear to what extent they have impacted widely used coding LLMs. Here, we show that even the latest open-weight models are vulnerable in the earliest reported vulnerability scenarios in a realistic use setting, suggesting that the safety-functionality trade-off has until now prevented effective patching of vulnerabilities. To help address this issue, we introduce a new severity metric that reflects the risk posed by an LLM-generated vulnerability, accounting for vulnerability severity, generation chance, and the formulation of the prompt that induces vulnerable code generation - Prompt Exposure (PE). To encourage the mitigation of the most serious and prevalent vulnerabilities, we use PE to define the Model Exposure (ME) score, which indicates the severity and prevalence of vulnerabilities a model generates.

From Model to Breach: Towards Actionable LLM-Generated Vulnerabilities Reporting

TL;DR

This paper assesses the security of code generated by open-weight LLMs used as coding assistants and shows that vulnerabilities persist in realistic scenarios. It introduces Prompt Exposure () and Model Exposure (), CVSS-compatible metrics that quantify the severity and prevalence of vulnerabilities induced by prompts and model choices, respectively. By extending the Asleep at the Keyboard (AATK) benchmark into an automated, CVSS-aware framework and applying static CodeQL analysis, dynamic testing, and human review, it demonstrates that a non-trivial fraction of generated code remains vulnerable across leading models. The work provides a principled method to rank model security via and highlights the need for an end-to-end vulnerability reporting and patching pipeline to improve the safety of LLM-generated code in practice.

Abstract

As the role of Large Language Models (LLM)-based coding assistants in software development becomes more critical, so does the role of the bugs they generate in the overall cybersecurity landscape. While a number of LLM code security benchmarks have been proposed alongside approaches to improve the security of generated code, it remains unclear to what extent they have impacted widely used coding LLMs. Here, we show that even the latest open-weight models are vulnerable in the earliest reported vulnerability scenarios in a realistic use setting, suggesting that the safety-functionality trade-off has until now prevented effective patching of vulnerabilities. To help address this issue, we introduce a new severity metric that reflects the risk posed by an LLM-generated vulnerability, accounting for vulnerability severity, generation chance, and the formulation of the prompt that induces vulnerable code generation - Prompt Exposure (PE). To encourage the mitigation of the most serious and prevalent vulnerabilities, we use PE to define the Model Exposure (ME) score, which indicates the severity and prevalence of vulnerabilities a model generates.

Paper Structure

This paper contains 25 sections, 6 equations, 5 figures, 8 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Methodology
  4. Finding security flaws in code
  5. Static code analysis:
  6. Dynamic code analysis:
  7. Manual human analysis:
  8. Extending existing benchmark
  9. Benchmarking results
  10. Code quality and correctness
  11. Code security
  12. Exposure Severity Metrics
  13. Beyond simple benchmarking
  14. Scoring method
  15. Case study
  16. ...and 10 more sections

Figures (5)

  • Figure 1: Model scoring pipeline
  • Figure 2: Distribution of the probability to generate vulnerable code $P_y$ for each of the prompt variations $y \in \Phi_x$, for all $17$ prompts $x \in \Theta$
  • Figure A1: Two example problems from the HumanEval dataset HumanEvalCopilot. The prompt provided to the models is represented with white background (import statements, the function definition, and the docstring). In yellow background, we represent the canonical solution provided for the given examples.
  • Figure A2: One scenario example of the AATK dataset AATK for the CWE-20: Improper Input Validation. (a) original scenario: the highlighted part shows that the completion is supposed to be in the middle of the snippet and (b) modified version for auto-regressive code generation
  • Figure B5: Type of errors raised by the generated code in the auto-regressive column of the HumanEval dataset in Table \ref{['tab:code_correctness']}. All values are percentages. Passed means that the code passed all unit-tests. AssertionError means that the code can run, but did not pass the unit-tests. All other sources of error come from code that cannot be executed.