Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agents

TL;DR

This work proposes ToolShield, a training-free, tool-agnostic, self-exploration defense that effectively reduces ASR by 30% on average in multi-turn interactions, and constructs MT-AgentRisk (Multi-Turn Agent Risk Benchmark), the first benchmark to evaluate multi-turn tool-using agent safety.

Abstract

LLM-based agents are becoming increasingly capable, yet their safety lags behind. This creates a gap between what agents can do and should do. This gap widens as agents engage in multi-turn interactions and employ diverse tools, introducing new risks overlooked by existing benchmarks. To systematically scale safety testing into multi-turn, tool-realistic settings, we propose a principled taxonomy that transforms single-turn harmful tasks into multi-turn attack sequences. Using this taxonomy, we construct MT-AgentRisk (Multi-Turn Agent Risk Benchmark), the first benchmark to evaluate multi-turn tool-using agent safety. Our experiments reveal substantial safety degradation: the Attack Success Rate (ASR) increases by 16% on average across open and closed models in multi-turn settings. To close this gap, we propose ToolShield, a training-free, tool-agnostic, self-exploration defense: when encountering a new tool, the agent autonomously generates test cases, executes them to observe downstream effects, and distills safety experiences for deployment. Experiments show that ToolShield effectively reduces ASR by 30% on average in multi-turn interactions. Our code is available at https://github.com/CHATS-lab/ToolShield.

Figures (44)

• Figure 1: As agents' capabilities grow, their safety falls behind, opening a widening capability-safety gap. To scale agent safety evaluation, we develop an attack taxonomy that transforms single-turn harmful tasks to multi-turn attack sequences. Applying the taxonomy, we construct MT-AgentRisk, the first agent safety benchmark in multi-turn, tool-realistic settings. To mitigate these risks, we propose ToolShield, a self-exploration defense that effectively protects tool-using agents in multi-turn interactions.
• Figure 2: The multi-turn attack taxonomy transforms a single-turn harmful task into an attack sequence. It operates along three dimensions: how the transformation is structured (Format), how it is performed (Method), and what is manipulated (Target). Transformation takes two main formats: Addition introduces additional layers to abstract the harm, while Decomposition fragments tasks into distributed subtasks reassembled later. Each format contains two methods. All transformation actions share a common target dimension (Data Files vs. Environment States), yielding 8 total subcategories. The examples show how A2 and D1 transform single-turn task to attack sequences.
• Figure 3: Distribution of turn counts in MT-AgentRisk. The average turns per task is 3.19, ranging from 2 to 7.
• Figure 4: Percentage of each subcategory in MT-AgentRisk.
• Figure 5: Pairwise cosine similarity of task-level text embeddings within each tool category and across all categories. All multi-turn instructions within a task are concatenated into a single document before embedding. Low similarity scores indicate diverse task coverage in MT-AgentRisk.