Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models

TL;DR

This work reveals that text-embedding outputs used in LLM safeguards concentrate in a biased region of the unit sphere and can be steered by universal magic words to manipulate text similarity. It introduces three search strategies—Brute-Force, Context-Free, and Gradient-Based—to efficiently discover single- and multi-token suffixes, and demonstrates jailbreaks on JailbreakBench, real-world chatbots, and cross-model/language transfer. The study also provides train-free defenses, notably renormalization, vocabulary cleaning, and reinitialization, that mitigate these attacks while improving downstream performance. The findings highlight a practical security risk in embedding-based safeguards and offer concrete mitigation strategies with broad implications for LLM safety and application domains.

Abstract

The security issue of large language models (LLMs) has gained wide attention recently, with various defense mechanisms developed to prevent harmful output, among which safeguards based on text embedding models serve as a fundamental defense. Through testing, we discover that the output distribution of text embedding models is severely biased with a large mean. Inspired by this observation, we propose novel, efficient methods to search for **universal magic words** that attack text embedding models. Universal magic words as suffixes can shift the embedding of any text towards the bias direction, thus manipulating the similarity of any text pair and misleading safeguards. Attackers can jailbreak the safeguards by appending magic words to user prompts and requiring LLMs to end answers with magic words. Experiments show that magic word attacks significantly degrade safeguard performance on JailbreakBench, cause real-world chatbots to produce harmful outputs in full-pipeline attacks, and generalize across input/output texts, models, and languages. To eradicate this security risk, we also propose defense methods against such attacks, which can correct the bias of text embeddings and improve downstream performance in a train-free manner.

Figures (25)

• Figure 1: The distribution of cosine similarity between text embedding $e(s)$ of text $s$ with normalized mean embedding $e^*$ of all text, tested on various text embedding models.
• Figure 2: Text embeddings concentrate in a band on the sphere $S^{d-1}$. Positive magic words can push them towards the normalized mean $e^*$. Negative magic words can pull them away from their original position.
• Figure 3: Pipeline to attack the safeguard of LLMs. The input guard is attacked directly by appending universal magic words to user prompts, and the output guard is indirectly attacked by requiring LLMs to append universal magic words to their output.
• Figure 4: The receiver operating characteristic (ROC) of input and output safeguards. Our magic words significantly reduce their area under the curve (AUC). Renormalization in the text embedding space mitigates the decrease of AUC and defends against this attack.
• Figure 5: Explicit attack failed.

Theorems & Definitions (4)

• Proposition 3.2
• Proposition 3.3
• proof
• proof