Smoke and Mirrors: Jailbreaking LLM-based Code Generation via Implicit Malicious Prompts
Sheng Ouyang, Yihao Qin, Bo Lin, Liqian Chen, Xiaoguang Mao, Shangwen Wang
TL;DR
This work exposes a vulnerability in LLM-based code generation by presenting CodeJailbreaker, which hides malicious intent in commit messages embedded within an otherwise benign prompt. The method is evaluated across three code-generation tasks on the RMCBench benchmark, showing high attack success and malicious-output rates for general LLMs, with variable resilience in code-specialized models. The findings illustrate a gap in current safety alignments, particularly against covert channels, and underscore the need for defenses that account for implicit cues in software evolution contexts. Overall, the paper highlights significant safety risks in code generation and provides a practical framework for assessing and mitigating implicit jailbreak strategies.
Abstract
The proliferation of Large Language Models (LLMs) has revolutionized natural language processing and significantly impacted code generation tasks, enhancing software development efficiency and productivity. Notably, LLMs like GPT-4 have demonstrated remarkable proficiency in text-to-code generation tasks. However, the growing reliance on LLMs for code generation necessitates a critical examination of the safety implications associated with their outputs. Existing research efforts have primarily focused on verifying the functional correctness of LLMs, overlooking their safety in code generation. This paper introduces a jailbreaking approach, CodeJailbreaker, designed to uncover safety concerns in LLM-based code generation. The basic observation is that existing safety mechanisms for LLMs are built through the instruction-following paradigm, where malicious intent is explicitly articulated within the instruction of the prompt. Consequently, CodeJailbreaker explores to construct a prompt whose instruction is benign and the malicious intent is implicitly encoded in a covert channel, i.e., the commit message, to bypass the safety mechanism. Experiments on the recently-released RMCBench benchmark demonstrate that CodeJailbreaker markedly surpasses the conventional jailbreaking strategy, which explicitly conveys malicious intents in the instructions, in terms of the attack effectiveness across three code generation tasks. This study challenges the traditional safety paradigms in LLM-based code generation, emphasizing the need for enhanced safety measures in safeguarding against implicit malicious cues.