DeceptPrompt: Exploiting LLM-driven Code Generation via Adversarial Natural Language Instructions
Fangzhou Wu, Xiaogeng Liu, Chaowei Xiao
TL;DR
This work examines the vulnerability of Code LLMs to adversarial natural-language prompts that preserve functional output while introducing targeted weaknesses. It introduces DeceptPrompt, an evolution-based framework that starts from benign seed prompts and, via a semantic-preserving genetic algorithm and a split-functionality/vulnerability loss, crafts prefixes/suffixes that coax code-generation models into producing vulnerable yet functional code. Across multiple Code LLMs and CWE-type vulnerabilities, DeceptPrompt achieves significant attack-success-rate gains and reveals systematic weaknesses in current systems, even under safety constraints. The findings motivate stronger alignment, safer prompt-design, and robust red-teaming practices for real-world deployment of code-generation LLMs.
Abstract
With the advancement of Large Language Models (LLMs), significant progress has been made in code generation, enabling LLMs to transform natural language into programming code. These Code LLMs have been widely accepted by massive users and organizations. However, a dangerous nature is hidden in the code, which is the existence of fatal vulnerabilities. While some LLM providers have attempted to address these issues by aligning with human guidance, these efforts fall short of making Code LLMs practical and robust. Without a deep understanding of the performance of the LLMs under the practical worst cases, it would be concerning to apply them to various real-world applications. In this paper, we answer the critical issue: Are existing Code LLMs immune to generating vulnerable code? If not, what is the possible maximum severity of this issue in practical deployment scenarios? In this paper, we introduce DeceptPrompt, a novel algorithm that can generate adversarial natural language instructions that drive the Code LLMs to generate functionality correct code with vulnerabilities. DeceptPrompt is achieved through a systematic evolution-based algorithm with a fine grain loss design. The unique advantage of DeceptPrompt enables us to find natural prefix/suffix with totally benign and non-directional semantic meaning, meanwhile, having great power in inducing the Code LLMs to generate vulnerable code. This feature can enable us to conduct the almost-worstcase red-teaming on these LLMs in a real scenario, where users are using natural language. Our extensive experiments and analyses on DeceptPrompt not only validate the effectiveness of our approach but also shed light on the huge weakness of LLMs in the code generation task. When applying the optimized prefix/suffix, the attack success rate (ASR) will improve by average 50% compared with no prefix/suffix applying.