Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters

Haibo Jin, Andy Zhou, Joe D. Menke, Haohan Wang

TL;DR

This work tackles the gap in evaluating moderation guardrails for LLMs by introducing JAMBench, a benchmark designed to trigger filtered-out errors in OpenAI-style moderation. It then proposes JAM, a jailbreak that fuses a jailbreak prefix with cipher characters to bypass input- and output-level safeguards. Across four LLMs, JAM achieves substantially higher jailbreak success ($\approx$ $19.88\times$) and lower filtering ($\approx$ $1/6$) than baselines, demonstrating the weakness of current guardrails against cipher-enabled prompts. The authors also propose two countermeasures—Output Complexity-Aware Defense and LLM-based Audit Defense—that can reduce jailbreak effectiveness to zero in tested scenarios, highlighting the need for stronger, more robust moderation mechanisms in practice.

Abstract

Large Language Models (LLMs) are typically harmless but remain vulnerable to carefully crafted prompts known as ``jailbreaks'', which can bypass protective measures and induce harmful behavior. Recent advancements in LLMs have incorporated moderation guardrails that can filter outputs, which trigger processing errors for certain malicious questions. Existing red-teaming benchmarks often neglect to include questions that trigger moderation guardrails, making it difficult to evaluate jailbreak effectiveness. To address this issue, we introduce JAMBench, a harmful behavior benchmark designed to trigger and evaluate moderation guardrails. JAMBench involves 160 manually crafted instructions covering four major risk categories at multiple severity levels. Furthermore, we propose a jailbreak method, JAM (Jailbreak Against Moderation), designed to attack moderation guardrails using jailbreak prefixes to bypass input-level filters and a fine-tuned shadow model functionally equivalent to the guardrail model to generate cipher characters to bypass output-level filters. Our extensive experiments on four LLMs demonstrate that JAM achieves higher jailbreak success ($\sim$ $\times$ 19.88) and lower filtered-out rates ($\sim$ $\times$ 1/6) than baselines.

Jailbreaking Large Language Models Against Moderation Guardrails via Cipher Characters

TL;DR

This work tackles the gap in evaluating moderation guardrails for LLMs by introducing JAMBench, a benchmark designed to trigger filtered-out errors in OpenAI-style moderation. It then proposes JAM, a jailbreak that fuses a jailbreak prefix with cipher characters to bypass input- and output-level safeguards. Across four LLMs, JAM achieves substantially higher jailbreak success ( ) and lower filtering ( ) than baselines, demonstrating the weakness of current guardrails against cipher-enabled prompts. The authors also propose two countermeasures—Output Complexity-Aware Defense and LLM-based Audit Defense—that can reduce jailbreak effectiveness to zero in tested scenarios, highlighting the need for stronger, more robust moderation mechanisms in practice.

Abstract

Large Language Models (LLMs) are typically harmless but remain vulnerable to carefully crafted prompts known as ``jailbreaks'', which can bypass protective measures and induce harmful behavior. Recent advancements in LLMs have incorporated moderation guardrails that can filter outputs, which trigger processing errors for certain malicious questions. Existing red-teaming benchmarks often neglect to include questions that trigger moderation guardrails, making it difficult to evaluate jailbreak effectiveness. To address this issue, we introduce JAMBench, a harmful behavior benchmark designed to trigger and evaluate moderation guardrails. JAMBench involves 160 manually crafted instructions covering four major risk categories at multiple severity levels. Furthermore, we propose a jailbreak method, JAM (Jailbreak Against Moderation), designed to attack moderation guardrails using jailbreak prefixes to bypass input-level filters and a fine-tuned shadow model functionally equivalent to the guardrail model to generate cipher characters to bypass output-level filters. Our extensive experiments on four LLMs demonstrate that JAM achieves higher jailbreak success ( 19.88) and lower filtered-out rates ( 1/6) than baselines.
Paper Structure (30 sections, 1 theorem, 11 equations, 4 figures, 10 tables, 1 algorithm)

This paper contains 30 sections, 1 theorem, 11 equations, 4 figures, 10 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Preliminaries
  5. Problem Definition
  6. Overview
  7. Construction of filtered corpus
  8. Construction of the shadow model
  9. Optimize cipher characters using jailbreak response format
  10. Generate the jailbreak prompt
  11. JAMBench
  12. Experiments
  13. Experimental Setup
  14. Effectiveness on Jailbreaking LLMs
  15. Ablation and Sensitivity Studies
  16. ...and 15 more sections

Key Result

Lemma 3.1

If $\dfrac{\partial \mathcal{L}^{adv}(\Tilde{\mathbf{x}}_{1:n})}{\partial \mathbf{x} }\dfrac{\partial \widehat{\mathcal{G}}(\mathbf{z};\theta) }{\partial \mathbf{x}} = \mathbf{0}$ for $\mathbf{x} \in \mathcal{A}(\hat{\mathbf{x}}_{1:n})$ and $\mathcal{A}(\hat{\mathbf{x}}_{1:n}) = \mathcal{A}_1(\hat{\

Figures (4)

  • Figure 1: Examples of jailbreaks. (a) A malicious question that receives a refusal response from the LLM. (b) An affirmative response with detailed steps to implement the malicious question by adding a jailbreak prompt as the prefix. (c) A filtered-out error is triggered by the moderation guardrail, even when a successful jailbreak prompt is added. (d) An affirmative response using JAM, which combines a jailbreak prefix, the malicious question, and the cipher characters to bypass the guardrail.
  • Figure 2: Three types of structural built-in safe guardrails.
  • Figure 3: Overview workflow of JAM for generating a jailbreak prompt, details in Section \ref{['overview']}.
  • Figure 4: Filtered-out rates of existing question benchmarks and JAMBench

Theorems & Definitions (2)

  • Lemma 3.1
  • proof