Jailbreaking Large Language Models in Infinitely Many Ways

TL;DR

The paper investigates Infinitely Many Paraphrases (IMP), a class of jailbreaking attacks that leverage a model's capacity to interpret paraphrases and encoded prompts to defeat safety guardrails. It formalizes a threat model with a guardrail $g$ and paraphrase set $\mathcal{X} \subset \sum^*$, showing that encoded prompts can bind semantics to English and bypass protections, a phenomenon studied through both scaling and inverse-scaling analyses. Empirically, IMPs bypass safety on a wide range of text and image models, including open- and closed-source systems, underscoring brittleness in several defenses such as the Anthropic Constitutional Classifier and DeepSeek. The authors propose token-space and embedding-space defenses, discuss multi-modal defenses, and outline future research directions and ethical considerations, calling for robust, scalable safety mechanisms that adapt to increasing model capabilities.

Abstract

We discuss the ``Infinitely Many Paraphrases'' attacks (IMP), a category of jailbreaks that leverages the increasing capabilities of a model to handle paraphrases and encoded communications to bypass their defensive mechanisms. IMPs' viability pairs and grows with a model's capabilities to handle and bind the semantics of simple mappings between tokens and work extremely well in practice, posing a concrete threat to the users of the most powerful LLMs in commerce. We show how one can bypass the safeguards of the most powerful open- and closed-source LLMs and generate content that explicitly violates their safety policies. One can protect against IMPs by improving the guardrails and making them scale with the LLMs' capabilities. For two categories of attacks that are straightforward to implement, i.e., bijection and encoding, we discuss two defensive strategies, one in token and the other in embedding space. We conclude with some research questions we believe should be prioritised to enhance the defensive mechanisms of LLMs and our understanding of their safety.

Figures (32)

• Figure 1: "Say 'hi' to the camera." Examples of satirical images the Infinitely Many Paraphrases attack allows to generate with a state-of-the-art LLM. The equivalent prompt in plain English, which includes the name of the person portrayed, is moderated and blocked by the model.
• Figure 2: An LLM is asked to generate an image of a person wearing a T-shirt. The model is asked to sketch "A cat" on a T-shirt; this instruction is encoded with a Caesar cipher. The conversation continues with the encoded instruction, "The cat wears glasses". In this case, the model decodes the input, as we do not ask it not to.
• Figure 3: Llama-3.1-405B-Instruct consistently handles conversations encoded with a cipher without explicitly decoding anything. Left: the encoded question is "What colour is the sky?", and the model correctly responds with "The color of the sky is blue." (encoded). Right: the encoded question is "What year did World War 2 end?", and the model correctly responds with "Yes, the World War 2 ended in 1945. It was a global conflict that lasted for many years." (encoded). For both the example, we used the Llama-3.1-405B-Instruct model hosted by https://sambanova.ai/.
• Figure 4: Two examples of fully functioning IMP as encoding/bijection prompts that bypass the guardrails of many state-of-the-art open- and closed-source LLMs. On top is an example that employs a routine in Haskell. Below is a mapping that uses Python dictionaries. They both encode the question "What is the capital of France?"
• Figure 5: We ask GPT-4o to draw a picture of a "black version of the head of the germany in the past century" (encoded). The attack employs a cipher (a paraphrase of an instruction that the guardrails would moderate) and a paraphrase the model implicitly understands.