Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysis

TL;DR

This work investigates jailbreaking of large language models through a representation-space lens, positing that successful attacks move harmful prompts toward the harmless prompt cluster along an acceptance direction in the model's latent space. It formalizes this idea by defining an anchored PCA space and an optimization objective that biases prompts to shift their representations along the acceptance direction, then demonstrates improvements in attack success for several white-box jailbreak methods on multiple models. The study also analyzes defenses, showing that paraphrasing can significantly disrupt the harmfulness signal while perplexity-based filters have model-dependent effects, and finds limited transferability of the enhanced attacks across models. Overall, the paper provides a representation-centric understanding of jailbreak dynamics and offers a practical objective to probe and potentially strengthen defenses against such attacks.

Abstract

Large language models (LLMs) are susceptible to a type of attack known as jailbreaking, which misleads LLMs to output harmful contents. Although there are diverse jailbreak attack strategies, there is no unified understanding on why some methods succeed and others fail. This paper explores the behavior of harmful and harmless prompts in the LLM's representation space to investigate the intrinsic properties of successful jailbreak attacks. We hypothesize that successful attacks share some similar properties: They are effective in moving the representation of the harmful prompt towards the direction to the harmless prompts. We leverage hidden representations into the objective of existing jailbreak attacks to move the attacks along the acceptance direction, and conduct experiments to validate the above hypothesis using the proposed objective. We hope this study provides new insights into understanding how LLMs understand harmfulness information.

Figures (6)

• Figure 1: Visualization of the representation from anchor prompts and jailbreak prompts with different attacks on different models: GCG (top), PAIR (middle) and AutoDAN (bottom). The shadowed eclipses represent the spread of each cluster at {1, 2, 3} standard variations, i.e., the regions $\{ \textbf{x}\in\mathbb{R}^2 | \textbf{x}^{T}\Sigma^{-1}\textbf{x}\leq a\}$ for $a\in\{1^2,2^2,3^2\}$ with $\textbf{x}=x_1x_2$ as values on the two coordinates, where $\Sigma$ is the covariance matrix of the corresponding cluster.
• Figure 2: Graphical illustration of early stopping.
• Figure 3: Visualization of representations of our methods on llama2-7b. The numerical distances are in Table \ref{['tab:ours_visualization_distance']}. This figure is drawn using the same method described in Section \ref{['sec:preliminary']}. It can be seen that, compared to the baseline method, our attack generally brings the jailbreak failed prompts and jailbreak succeeded prompts closer to the acceptance center.
• Figure 4: Full result of visualization of the representations, without early stopping on GCG and AutoDAN.
• Figure 5: Randomly chosen loss curves from GCG(+Ours) and AutoDAN(+Ours) on gemma-7b. The negative loss values are a result of taking the negative of the loss function to convert a maximization problem into a minimization problem. One can observe that the loss fluctuation of AutoDAN is more drastic, and to some extent exhibits the characteristics of random searching, while the decrease in loss may play a smaller role compared to GCG.

Theorems & Definitions (1)

• Definition 3.1