Prompt Injection Attack to Tool Selection in LLM Agents

TL;DR

ToolHijacker reveals a vulnerability in LLM agent tool selection by injecting malicious tool documents in a no-box setting. It formalizes a shadowed optimization framework with two phases to manipulate both retrieval and selection, and develops gradient-free and gradient-based methods to craft R and S components of the malicious tool description. Extensive evaluations across multiple datasets, LLMs, and retrievers show high attack success and retrieval impact, while existing prevention and detection defenses largely fail. The work highlights the urgent need for novel defense mechanisms to secure tool ecosystems around LLM agents and suggests directions for extending the attack surface and defenses in future research.

Abstract

Tool selection is a key component of LLM agents. A popular approach follows a two-step process - \emph{retrieval} and \emph{selection} - to pick the most appropriate tool from a tool library for a given task. In this work, we introduce \textit{ToolHijacker}, a novel prompt injection attack targeting tool selection in no-box scenarios. ToolHijacker injects a malicious tool document into the tool library to manipulate the LLM agent's tool selection process, compelling it to consistently choose the attacker's malicious tool for an attacker-chosen target task. Specifically, we formulate the crafting of such tool documents as an optimization problem and propose a two-phase optimization strategy to solve it. Our extensive experimental evaluation shows that ToolHijacker is highly effective, significantly outperforming existing manual-based and automated prompt injection attacks when applied to tool selection. Moreover, we explore various defenses, including prevention-based defenses (StruQ and SecAlign) and detection-based defenses (known-answer detection, DataSentinel, perplexity detection, and perplexity windowed detection). Our experimental results indicate that these defenses are insufficient, highlighting the urgent need for developing new defense strategies.

Figures (16)

• Figure 1: Illustration of tool selection in LLM agents under no attack and our attack.
• Figure 2: Illustration of Step 2 - Selection.
• Figure 3: Our attacks are effective across different tasks.
• Figure 4: Token length of benign tool documents and malicious tool documents generated via different attacks.
• Figure 5: AHRs and ASRs with different $k'$ of the shadow retriever and $k$ of the target retriever.