Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Jailbreaking Attacks vs. Content Safety Filters: How Far Are We in the LLM Safety Arms Race?

Yuan Xin, Dingfan Chen, Linyi Yang, Michael Backes, Xiao Zhang

TL;DR

This paper investigates whether content safety filters can effectively block jailbreak attacks in LLM deployment pipelines, beyond model-level defenses. It formalizes a measurement framework that combines an LLM, a harm-detection judge, and various safety filters, and evaluates a curated dataset of 417 harmful prompts with benign counterparts across 10 categories on multiple open-weight and commercial LLMs using 10 jailbreak techniques. The key findings show that safety filters substantially reduce attack success in practice, with input-stage detectors often achieving high recall and overall pass rates typically under 5%, though there is a notable trade-off with false positives and usability. The work highlights variations in detector performance across models and attack types, demonstrates the practicality of system-level defenses, and calls for ongoing refinement of detection accuracy, efficiency, and broader, community-driven benchmarking to strengthen LLM safety in real-world deployments.

Abstract

As large language models (LLMs) are increasingly deployed, ensuring their safe use is paramount. Jailbreaking, adversarial prompts that bypass model alignment to trigger harmful outputs, present significant risks, with existing studies reporting high success rates in evading common LLMs. However, previous evaluations have focused solely on the models, neglecting the full deployment pipeline, which typically incorporates additional safety mechanisms like content moderation filters. To address this gap, we present the first systematic evaluation of jailbreak attacks targeting LLM safety alignment, assessing their success across the full inference pipeline, including both input and output filtering stages. Our findings yield two key insights: first, nearly all evaluated jailbreak techniques can be detected by at least one safety filter, suggesting that prior assessments may have overestimated the practical success of these attacks; second, while safety filters are effective in detection, there remains room to better balance recall and precision to further optimize protection and user experience. We highlight critical gaps and call for further refinement of detection accuracy and usability in LLM safety systems.

Jailbreaking Attacks vs. Content Safety Filters: How Far Are We in the LLM Safety Arms Race?

TL;DR

This paper investigates whether content safety filters can effectively block jailbreak attacks in LLM deployment pipelines, beyond model-level defenses. It formalizes a measurement framework that combines an LLM, a harm-detection judge, and various safety filters, and evaluates a curated dataset of 417 harmful prompts with benign counterparts across 10 categories on multiple open-weight and commercial LLMs using 10 jailbreak techniques. The key findings show that safety filters substantially reduce attack success in practice, with input-stage detectors often achieving high recall and overall pass rates typically under 5%, though there is a notable trade-off with false positives and usability. The work highlights variations in detector performance across models and attack types, demonstrates the practicality of system-level defenses, and calls for ongoing refinement of detection accuracy, efficiency, and broader, community-driven benchmarking to strengthen LLM safety in real-world deployments.

Abstract

As large language models (LLMs) are increasingly deployed, ensuring their safe use is paramount. Jailbreaking, adversarial prompts that bypass model alignment to trigger harmful outputs, present significant risks, with existing studies reporting high success rates in evading common LLMs. However, previous evaluations have focused solely on the models, neglecting the full deployment pipeline, which typically incorporates additional safety mechanisms like content moderation filters. To address this gap, we present the first systematic evaluation of jailbreak attacks targeting LLM safety alignment, assessing their success across the full inference pipeline, including both input and output filtering stages. Our findings yield two key insights: first, nearly all evaluated jailbreak techniques can be detected by at least one safety filter, suggesting that prior assessments may have overestimated the practical success of these attacks; second, while safety filters are effective in detection, there remains room to better balance recall and precision to further optimize protection and user experience. We highlight critical gaps and call for further refinement of detection accuracy and usability in LLM safety systems.
Paper Structure (19 sections, 6 equations, 7 figures, 11 tables)

This paper contains 19 sections, 6 equations, 7 figures, 11 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Our Measurement Framework
  4. Problem Formulation
  5. Safety Filters
  6. Experimental Setup
  7. Experiments
  8. Safety Filters vs. Jailbreak Attacks
  9. Analysis of Detector's Behavior
  10. Qualitative Demonstrations
  11. Conclusion
  12. Additional Details
  13. Experimental Setup
  14. System Prompts
  15. Content Safety Filters
  16. ...and 4 more sections

Figures (7)

  • Figure 1: Detection rates on samples judged as attack-successful by $\mathsf{Judge}$ against Mistral-7B on our curated dataset. (a) detection rate on input level; (b) detection rate on output level.
  • Figure 2: Detection TP, TN, FP, FN rates on "normal" (including both "benign" and "harmful") samples.
  • Figure 3: Input-stage detection ("Detected"/"Undetected") vs. attack ("Success"/"Fail") rates on Mistral-7B.
  • Figure 4: Distribution of categories for false positive benign behaviors by safety filters.
  • Figure 5: Input-stage detection ("Detected"/"Undetected") vs. attack ("Success"/"Fail") rates on Mistral-7B. "Deepincept" is short for Deepinception,"Code" is short for CodeChameleon.
  • ...and 2 more figures