David vs. Goliath: Verifiable Agent-to-Agent Jailbreaking via Reinforcement Learning

TL;DR

This paper formalizes Tag-Along Attacks, a verifiable two-agent jailbreak threat where a smaller adversary exploits an Operator's tool privileges through conversation. It introduces Slingshot, a cold-start reinforcement learning framework trained with CISPO to autonomously discover short, instruction-like attack vectors, and TagAlong-Dojo, a 575-task benchmark for agent-to-agent red-teaming. Results show high attack success on extreme tasks (ASR around 67.0%) and strong zero-shot transfer to both open and closed models, revealing brittle guardrails and an alignment tax when defenses are narrowly targeted. The work argues for viewing agent safety as a verifiable control problem and advocates automated, optimization-driven stress testing to build robust defenses, while releasing the attack framework and benchmark to support reproducible safety research.

Abstract

The evolution of large language models into autonomous agents introduces adversarial failures that exploit legitimate tool privileges, transforming safety evaluation in tool-augmented environments from a subjective NLP task into an objective control problem. We formalize this threat model as Tag-Along Attacks: a scenario where a tool-less adversary "tags along" on the trusted privileges of a safety-aligned Operator to induce prohibited tool use through conversation alone. To validate this threat, we present Slingshot, a 'cold-start' reinforcement learning framework that autonomously discovers emergent attack vectors, revealing a critical insight: in our setting, learned attacks tend to converge to short, instruction-like syntactic patterns rather than multi-turn persuasion. On held-out extreme-difficulty tasks, Slingshot achieves a 67.0% success rate against a Qwen2.5-32B-Instruct-AWQ Operator (vs. 1.7% baseline), reducing the expected attempts to first success (on solved tasks) from 52.3 to 1.3. Crucially, Slingshot transfers zero-shot to several model families, including closed-source models like Gemini 2.5 Flash (56.0% attack success rate) and defensive-fine-tuned open-source models like Meta-SecAlign-8B (39.2% attack success rate). Our work establishes Tag-Along Attacks as a first-class, verifiable threat model and shows that effective agentic attacks can be elicited from off-the-shelf open-weight models through environment interaction alone.

Figures (5)

• Figure 1: The Slingshot Framework. The attacker $\mathcal{S}$ engages the safety-aligned Operator $\mathcal{O}$ in a conversation to execute a malicious task $\tau$. Unlike subjective jailbreaking, success is determined by the environment $\mathcal{E}$ verifying if the prohibited tool sequence was executed in the latent state $\Sigma$ (i.e., $s_{\mathcal{E}}(\tau)=1$).
• Figure 2: Extreme-task performance.Slingshot increases ASR and Pass@10 while reducing refusals, and cuts attempts-to-success from 52.3 to 1.3 on average, compared to the Base-A baseline.
• Figure 3: Token usage with vs. without the gibberish penalty. With the penalty, attacks remain short; without it, the policy often length-maximizes to the 1024-token cap, primarily via repetition.
• Figure 4: Representative degenerate repetition when disabling the gibberish penalty. The policy reaches the 1024-token cap largely by repeating an imperative demand, rather than adding new planning content.
• Figure 5: Judge shaping ablation on the held-out extreme split.Top: held-out extreme-split ASR during training, with vs. without judge shaping (rolling window size 5). Bottom: final checkpoint metrics (41 extreme tasks, 100 attempts/task), evaluated with a single-turn 1024-token attacker budget. Without judge shaping sets $R_{\text{shape}}\equiv 0$ while keeping all other reward terms fixed (including $R_{\text{success}}$ and quit/refusal penalties). Note: the With judge shaping numbers use our main multi-turn-trained checkpoint, while the Without judge shaping run was trained single-turn; this should not affect the interpretation because the main-text multi-turn vs. single-turn ablation shows comparable single-turn evaluation performance.