Evaluating the Robustness of Large Language Model Safety Guardrails Against Adversarial Attacks

TL;DR

This work critically evaluates the robustness of large language model safety guardrails against adversarial prompts, revealing that high benchmark performance often fails to generalize to novel attacks. By testing ten guardrail models across 1,445 prompts in 21 attack categories, the study exposes large generalization gaps, including a 57.2-point drop for the top performer when faced with unseen prompts. It also uncovers a dangerous "helpful mode" jailbreak where some guardrails generate harmful content instead of refusing, highlighting vulnerabilities beyond traditional accuracy metrics. The findings stress the need for evaluation protocols that prioritize generalization, benchmark transparency, and defense-in-depth in real-world deployments.

Abstract

Large Language Model (LLM) safety guardrail models have emerged as a primary defense mechanism against harmful content generation, yet their robustness against sophisticated adversarial attacks remains poorly characterized. This study evaluated ten publicly available guardrail models from Meta, Google, IBM, NVIDIA, Alibaba, and Allen AI across 1,445 test prompts spanning 21 attack categories. While Qwen3Guard-8B achieved the highest overall accuracy (85.3%, 95% CI: 83.4-87.1%), a critical finding emerged when separating public benchmark prompts from novel attacks: all models showed substantial performance degradation on unseen prompts, with Qwen3Guard dropping from 91.0% to 33.8% (a 57.2 percentage point gap). In contrast, Granite-Guardian-3.2-5B showed the best generalization with only a 6.5% gap. A "helpful mode" jailbreak was also discovered where two guardrail models (Nemotron-Safety-8B, Granite-Guardian-3.2-5B) generated harmful content instead of blocking it, representing a novel failure mode. These findings suggest that benchmark performance may be misleading due to training data contamination, and that generalization ability, not overall accuracy, should be the primary metric for guardrail evaluation.

Figures (9)

• Figure 1: Hierarchical taxonomy of adversarial test prompts. Tree diagram showing the organization of 1,445 test prompts across 12 attack categories. Public benchmarks (red branches, n=1,300) include JailbreakBench jailbreaks and TrustAIRLab wild prompts, achieving 54--91% detection. Novel prompts (green branches, n=145) were created for this study using professional framing strategies: business attacks (corporate scenarios), advanced attacks (research contexts), subtle adversarial (social engineering), overt adversarial (direct overrides), complex adversarial (multi-step), and boundary cases. Novel categories achieve only 3--50% detection, revealing systematic blind spots in current guardrails.
• Figure 2: Evaluation methodology pipeline. Flowchart illustrating the experimental procedure from data collection through analysis. (a) Input stage: test prompts from public benchmarks and novel sources are combined with model-specific system prompts. (b) Inference stage: each prompt is processed by 10 guardrail models with standardized parameters (temperature=0.1, max_tokens=128). (c) Parsing stage: hierarchical rule-based classifier handles diverse output formats including binary verdicts, tagged scores, and labeled outputs. (d) Detection of "helpful mode" failures where models generate substantive content instead of safety classifications. (e) Final aggregation computes accuracy metrics, confidence intervals, and statistical comparisons.
• Figure 3: Overall guardrail model performance on adversarial prompts. Horizontal bars show classification accuracy across 1,445 test prompts spanning 12 attack categories. Colors indicate model provider (see legend). Vertical dashed lines mark performance tiers: chance level (50%) and strong performance (80%). Three models---Qwen3Guard-8B (Alibaba), WildGuard-7B (Allen AI), and Granite-Guardian-3.3-8B (IBM)---exceed the 80% threshold. The 2.2$\times$ performance gap between best (85.3%) and worst (38.0%) models indicates substantial heterogeneity in guardrail effectiveness. Note: LlamaGuard-7B* is a community fine-tune, not an official Meta release.
• Figure 4: Safety versus usability trade-off in guardrail models. Scatter plot showing benign accuracy (x-axis, correctly allowing safe prompts) versus harmful accuracy (y-axis, correctly blocking dangerous prompts) for each model. The ideal zone (upper-right quadrant) requires high performance on both dimensions. LlamaGuard variants cluster in the "too permissive" region (lower-right): they achieve 97--99% benign accuracy but only 4.5--21.8% harmful detection, effectively rubber-stamping most inputs. Qwen3Guard-8B, WildGuard-7B, and Granite-Guardian-3.3-8B approach balanced performance in the upper-right quadrant. Colors indicate model provider.
• Figure 5: Generalization gap reveals potential benchmark overfitting. Grouped bar chart comparing model accuracy on public benchmark prompts (blue, n=1,300 from JailbreakBench, TrustAIRLab, etc.) versus private novel prompts (red, n=145 created for this study). Models are sorted by generalization ability (smallest gap = best). Red annotations show accuracy degradation. Critically, Qwen3Guard-8B---despite achieving highest overall accuracy---shows the largest gap ($-$57.2%), dropping from 91% on public benchmarks to just 34% on novel prompts. This pattern suggests overfitting to public benchmark distributions or potential training data overlap. Granite-Guardian-3.2-5B generalizes best ($-$6.3%), likely indicating more diverse training data. Dashed line indicates random baseline (50%).