Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Exploiting Prefix-Tree in Structured Output Interfaces for Enhancing Jailbreak Attacking

Yanzeng Li, Yunfan Xiong, Jialun Zhong, Jinchao Zhang, Jie Zhou, Lei Zou

TL;DR

The study identifies a new vulnerability surface for LLMs: structured output interfaces can enable jailbreak attacks by manipulating token-level sampling while safety patterns are learned at the sentence level. It introduces AttackPrefixTree (APT), a black-box two-phase framework that exploits constrained decoding and safety prefixes to generate harmful content, demonstrated across HarmBench, AdvBench, and JailbreakBench with strong ASR improvements. Phase 1 constructs a prefix tree of safe and harmful fragments via DFS, while Phase 2 reranks paths using a discriminator to select the most harmful output, all within a constrained decoding environment. The findings emphasize the need for enhanced defenses—such as real-time monitoring of constrained decoding, diversified refusal templates, and multi-stage safety alignment—to secure structured-output capabilities in LLM services against evolving jailbreak threats.

Abstract

The rise of Large Language Models (LLMs) has led to significant applications but also introduced serious security threats, particularly from jailbreak attacks that manipulate output generation. These attacks utilize prompt engineering and logit manipulation to steer models toward harmful content, prompting LLM providers to implement filtering and safety alignment strategies. We investigate LLMs' safety mechanisms and their recent applications, revealing a new threat model targeting structured output interfaces, which enable attackers to manipulate the inner logit during LLM generation, requiring only API access permissions. To demonstrate this threat model, we introduce a black-box attack framework called AttackPrefixTree (APT). APT exploits structured output interfaces to dynamically construct attack patterns. By leveraging prefixes of models' safety refusal response and latent harmful outputs, APT effectively bypasses safety measures. Experiments on benchmark datasets indicate that this approach achieves higher attack success rate than existing methods. This work highlights the urgent need for LLM providers to enhance security protocols to address vulnerabilities arising from the interaction between safety patterns and structured outputs.

Exploiting Prefix-Tree in Structured Output Interfaces for Enhancing Jailbreak Attacking

TL;DR

The study identifies a new vulnerability surface for LLMs: structured output interfaces can enable jailbreak attacks by manipulating token-level sampling while safety patterns are learned at the sentence level. It introduces AttackPrefixTree (APT), a black-box two-phase framework that exploits constrained decoding and safety prefixes to generate harmful content, demonstrated across HarmBench, AdvBench, and JailbreakBench with strong ASR improvements. Phase 1 constructs a prefix tree of safe and harmful fragments via DFS, while Phase 2 reranks paths using a discriminator to select the most harmful output, all within a constrained decoding environment. The findings emphasize the need for enhanced defenses—such as real-time monitoring of constrained decoding, diversified refusal templates, and multi-stage safety alignment—to secure structured-output capabilities in LLM services against evolving jailbreak threats.

Abstract

The rise of Large Language Models (LLMs) has led to significant applications but also introduced serious security threats, particularly from jailbreak attacks that manipulate output generation. These attacks utilize prompt engineering and logit manipulation to steer models toward harmful content, prompting LLM providers to implement filtering and safety alignment strategies. We investigate LLMs' safety mechanisms and their recent applications, revealing a new threat model targeting structured output interfaces, which enable attackers to manipulate the inner logit during LLM generation, requiring only API access permissions. To demonstrate this threat model, we introduce a black-box attack framework called AttackPrefixTree (APT). APT exploits structured output interfaces to dynamically construct attack patterns. By leveraging prefixes of models' safety refusal response and latent harmful outputs, APT effectively bypasses safety measures. Experiments on benchmark datasets indicate that this approach achieves higher attack success rate than existing methods. This work highlights the urgent need for LLM providers to enhance security protocols to address vulnerabilities arising from the interaction between safety patterns and structured outputs.

Paper Structure

This paper contains 19 sections, 4 equations, 3 figures, 4 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Preliminary & Related Works
  3. Structured Output
  4. LLM Jailbreak
  5. Auto-method
  6. Anti-Jailbreak Techniques
  7. Methodology
  8. Threat Model
  9. Implementation Details
  10. Experiments
  11. Setting
  12. Experimental Results
  13. Parameter Analysis
  14. Attack on Reasoning Models
  15. Conclusion and Discussion
  16. ...and 4 more sections

Figures (3)

  • Figure 1: An example of constrained decoding. The constrained pattern in Regex is (?!Sorry|It's illegal).*.
  • Figure 2: The overall diagram of our framework.
  • Figure 3: Multiple LLMs' ASR of JailbreakBench across different beam sizes.