LLMSecEval: A Dataset of Natural Language Prompts for Security Evaluations

TL;DR

The paper tackles the risk of insecure code produced by NL-driven LLM code generation by introducing LLMSecEval, a dataset of 150 NL prompts aligned to 18 CWE weaknesses, each paired with secure implementations. It outlines a pipeline that starts from Pearce Copilot data, uses Codex to translate code to NL prompts, and applies manual curation before producing a usable benchmark. The authors provide a quality analysis of prompts, an end-to-end demonstration application combining GPT-3/Codex with CodeQL, and publicly release the dataset to enable reproducible security testing and prompt-engineering for secure code generation. This work enables systematic evaluation of NL-based code generation for security, offering concrete prompts, secure baselines, and tooling to drive future improvements in prompt design and LLM safety.

Abstract

Large Language Models (LLMs) like Codex are powerful tools for performing code completion and code generation tasks as they are trained on billions of lines of code from publicly available sources. Moreover, these models are capable of generating code snippets from Natural Language (NL) descriptions by learning languages and programming practices from public GitHub repositories. Although LLMs promise an effortless NL-driven deployment of software applications, the security of the code they generate has not been extensively investigated nor documented. In this work, we present LLMSecEval, a dataset containing 150 NL prompts that can be leveraged for assessing the security performance of such models. Such prompts are NL descriptions of code snippets prone to various security vulnerabilities listed in MITRE's Top 25 Common Weakness Enumeration (CWE) ranking. Each prompt in our dataset comes with a secure implementation example to facilitate comparative evaluations against code produced by LLMs. As a practical application, we show how LLMSecEval can be used for evaluating the security of snippets automatically generated from NL descriptions.

Figures (3)

• Figure 1: NL prompts creation process
• Figure 2: An example of NL prompt generated from a Python code snippet covering CWE-20 scenario in the Pearce et al. PearceA0DK22 dataset.
• Figure 3: Language- and content-related scores (Note: Frequencies lower than 2 are not labeled in the graph).