CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completion

TL;DR

CodeAttack presents a framework to assess safety generalization of LLMs by converting natural-language prompts into a code completion task. It reveals that current safety alignments struggle to generalize to code-based inputs, with attack success rates exceeding 80% on several models. The study analyzes how input encoding, task understanding, and output specification shape vulnerability, and highlights the influence of code-domain biases and language distribution. It also evaluates defenses, finding limited effectiveness and underscoring the need for robust, domain-aware safety alignment and red-teaming.

Abstract

The rapid advancement of Large Language Models (LLMs) has brought about remarkable generative capabilities but also raised concerns about their potential misuse. While strategies like supervised fine-tuning and reinforcement learning from human feedback have enhanced their safety, these methods primarily focus on natural languages, which may not generalize to other domains. This paper introduces CodeAttack, a framework that transforms natural language inputs into code inputs, presenting a novel environment for testing the safety generalization of LLMs. Our comprehensive studies on state-of-the-art LLMs including GPT-4, Claude-2, and Llama-2 series reveal a new and universal safety vulnerability of these models against code input: CodeAttack bypasses the safety guardrails of all models more than 80\% of the time. We find that a larger distribution gap between CodeAttack and natural language leads to weaker safety generalization, such as encoding natural language input with data structures. Furthermore, we give our hypotheses about the success of CodeAttack: the misaligned bias acquired by LLMs during code training, prioritizing code completion over avoiding the potential safety risk. Finally, we analyze potential mitigation measures. These findings highlight new safety risks in the code domain and the need for more robust safety alignment algorithms to match the code capabilities of LLMs.

Figures (8)

• Figure 1: Overview of our CodeAttack. CodeAttack constructs a code template with three steps: (1) Input encoding which encodes the harmful text-based query with common data structures; (2) Task understanding which applies a $decode()$ function to allow LLMs to extract the target task from various kinds of inputs; (3) Output specification which enables LLMs to fill the output structure with the user's desired content.
• Figure 2: Example of different data structures for input encoding in CodeAttack in Python environment. The types of data structure and the initialization way decide the similarity of the encoded input to natural language. We select Python string to encapsulate the entire natural language query. Besides string, we utilize Python list and deque to represent the data structure queue and stack respectively.
• Figure 3: Examples of the ablation study for task understanding in CodeAttack. We crop the code snippet for better visualization.
• Figure 4: Examples of the ablation study for output specification in CodeAttack. We crop the code snippet for better visualization.
• Figure 5: An example of our CodeAttack and the responses of GPT-4, Claude-2, and Llama-2-70b-chat. CodeAttack is implemented in Python and the input is encoded into a Python list.