STACK: Adversarial Attacks on LLM Safeguard Pipelines

TL;DR

This work analyzes defense-in-depth pipelines guarding frontier LLMs and introduces STACK, a staged attack that targets input and output safeguards. It demonstrates that open-weight safeguard models can be significantly weakened, with Gemma2 outperforming existing options on harmful datasets. The authors show that a black-box front-to-back STACK achieves $71 ext{ ext{%}}$ ASR on ClearHarm and $33 ext{ ext{%}}$ ASR under transfer, underscoring transfer viability even without direct pipeline access. The paper concludes with practical mitigations to harden pipelines, emphasizing stronger, diverse safeguards and reducing information leakage to thwart staged, component-wise jailbreaks.

Abstract

Frontier AI developers are relying on layers of safeguards to protect against catastrophic misuse of AI systems. Anthropic and OpenAI guard their latest Opus 4 model and GPT-5 models using such defense pipelines, and other frontier developers including Google DeepMind pledge to soon deploy similar defenses. However, the security of such pipelines is unclear, with limited prior work evaluating or attacking these pipelines. We address this gap by developing and red-teaming an open-source defense pipeline. First, we find that a novel few-shot-prompted input and output classifier outperforms state-of-the-art open-weight safeguard model ShieldGemma across three attacks and two datasets, reducing the attack success rate (ASR) to 0% on the catastrophic misuse dataset ClearHarm. Second, we introduce a STaged AttaCK (STACK) procedure that achieves 71% ASR on ClearHarm in a black-box attack against the few-shot-prompted classifier pipeline. Finally, we also evaluate STACK in a transfer setting, achieving 33% ASR, providing initial evidence that it is feasible to design attacks with no access to the target pipeline. We conclude by suggesting specific mitigations that developers could use to thwart staged attacks.

Figures (21)

• Figure 1: Left: Sketch of the STACK attack, with the query containing the input classifier jailbreak and a request to repeat the output classifier jailbreak. Right: Attack success rate (ASR) of SOTA black-box attack PAPpap vs. STACK (ours). STACK (-- -- --) does worse against the undefended model than PAP (-- -- --), but makes up for it by defeating the classifier s (-----), leading to an overall ASR of 71% (compared to 0% for PAP , -----).
• Figure 2: Overview of defense-in-depth threat models. The two axes are separability of the pipeline---how much access the attacker gets to each defense component---and opacity of the components---whether the attacker gets full access to the weights (white-box) or just boolean outputs (black-box). The main threat model we consider is semi-separable black-box access, where the attacker runs a query on the pipeline, and can see which component blocked the query. For transfer, we assume separable white-box access to a proxy pipeline and inseparable black-box access to the target pipeline. For more details, see Appendix \ref{['app:threat-models']}.
• Figure 3: Attack success rate (ASR) achieved against the undefended Qwen3-14B model ("Undef.") and each defense pipeline against PAP , ReNeLLM, and Best-of-N on StrongREJECT and ClearHarm. Details of the defense pipelines are given in Appendix \ref{['app:defense-pipelines']}, and exact results in Appendix \ref{['sec:asr_results']}.
• Figure 4: Attack Success Rate (ASR) of PAP and front-to-back STACK on the ClearHarm dataset against the "Gemma2" pipeline. (A) through (D) are different seeds of STACK, based on different input and output jailbreaks. The best seed (A) effectively bypasses the safeguards of the "Defended" model (-----), getting close to the "Undefended" (generative model only, -- -- --) ASR. However, STACK damages the ability of PAP to jailbreak the generative model (discussed more in Section \ref{['sec:limitations-future-work']}), performing worse than vanilla PAP (-- -- --) on "Undefended". The $x$-axis shows how the ASR increases with the PAP iterations applied to rephrase each datapoint, with input and output jailbreaks fixed. See Appendix \ref{['app:other-models']} for the results on other models and datasets.
• Figure 5: Prompt template supplied to the judge.