Inducing Vulnerable Code Generation in LLM Coding Assistants

TL;DR

Hackode reveals a new threat where LLM coding assistants referencing externally posted solutions can be steered to emit vulnerable code via hidden attack sequences. It uses a two-phase, gradient-guided attack sequence generation that adapts to diverse assembled inputs formed by varying prompt templates, instructions, and user queries, achieving an average ASR of $84.29\%$ across four open-source LLMs and $75.92\%$ in a real-world setup. Key contributions include identifying a practical external-information attack vector, constructing a transferable attack pipeline, and demonstrating substantial transferability and real-world impact through extensive experiments and ablation studies. The work underscores the need for defenses combining static/dynamic analysis and runtime monitoring to mitigate vulnerabilities propagated through externally-referenced code.

Abstract

Due to insufficient domain knowledge, LLM coding assistants often reference related solutions from the Internet to address programming problems. However, incorporating external information into LLMs' code generation process introduces new security risks. In this paper, we reveal a real-world threat, named HACKODE, where attackers exploit referenced external information to embed attack sequences, causing LLMs to produce code with vulnerabilities such as buffer overflows and incomplete validations. We designed a prototype of the attack, which generates effective attack sequences for potential diverse inputs with various user queries and prompt templates. Through the evaluation on two general LLMs and two code LLMs, we demonstrate that the attack is effective, achieving an 84.29% success rate. Additionally, on a real-world application, HACKODE achieves 75.92% ASR, demonstrating its real-world impact.

Figures (5)

• Figure 1: An example of Hackode on an LLM coding assistant. The assistant references the correct code posted by attackers, leading LLMs to generate code with vulnerability.
• Figure 2: An example of assembled input derivation. Attackers produce an assembled input by randomly combining a query, an instruction, and the reference information according to a prompt template.
• Figure 3: An example of Hackode. The attacker first writes a correct code solution for the unanswered question. The attacker then makes subtle modifications to the correct code to introduce vulnerabilities $tVul$. After the attack sequence generation, the attacker injects the attack sequence into insertion positions within the correct code and uses this to induce the LLM to generate code that contains vulnerabilities $tVul$.
• Figure 4: Overview of Hackode: First, it derives various assembled inputs by considering different instructions, prompt templates, and user queries. Second, it generates a preliminary attack sequence for a fixed assembled input. Third, it refines the sequence to enhance its transferability across the derived diverse assembled inputs.
• Figure 5: A real-world example of Hackode. Attackers post a correct solution with an attack sequence, guiding the coding assistant to generate vulnerable code with 'out-of-bounds write'.