Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

From Allies to Adversaries: Manipulating LLM Tool-Calling through Adversarial Injection

Haowei Wang, Rupeng Zhang, Junjie Wang, Mingyang Li, Yuekai Huang, Dandan Wang, Qing Wang

TL;DR

ToolCommander reveals critical security vulnerabilities in LLM tool-calling by injecting adversarial Manipulator Tools that harvest real user queries and disrupt tool scheduling, enabling privacy theft, denial-of-service, and unscheduled tool calling. The two-stage approach—query collection followed by scheduling manipulation—achieves high attack success across multiple LLMs and retrievers, with both white-box and black-box configurations analyzed. The work provides a rigorous evaluation framework, demonstrates superior execution over baselines, and highlights the need for defense-centric tool-augmentation strategies. It underscores practical risks in real-world deployments and offers guidance for designing robust, secure tool-enabled LLM systems.

Abstract

Tool-calling has changed Large Language Model (LLM) applications by integrating external tools, significantly enhancing their functionality across diverse tasks. However, this integration also introduces new security vulnerabilities, particularly in the tool scheduling mechanisms of LLM, which have not been extensively studied. To fill this gap, we present ToolCommander, a novel framework designed to exploit vulnerabilities in LLM tool-calling systems through adversarial tool injection. Our framework employs a well-designed two-stage attack strategy. Firstly, it injects malicious tools to collect user queries, then dynamically updates the injected tools based on the stolen information to enhance subsequent attacks. These stages enable ToolCommander to execute privacy theft, launch denial-of-service attacks, and even manipulate business competition by triggering unscheduled tool-calling. Notably, the ASR reaches 91.67% for privacy theft and hits 100% for denial-of-service and unscheduled tool calling in certain cases. Our work demonstrates that these vulnerabilities can lead to severe consequences beyond simple misuse of tool-calling systems, underscoring the urgent need for robust defensive strategies to secure LLM Tool-calling systems.

From Allies to Adversaries: Manipulating LLM Tool-Calling through Adversarial Injection

TL;DR

ToolCommander reveals critical security vulnerabilities in LLM tool-calling by injecting adversarial Manipulator Tools that harvest real user queries and disrupt tool scheduling, enabling privacy theft, denial-of-service, and unscheduled tool calling. The two-stage approach—query collection followed by scheduling manipulation—achieves high attack success across multiple LLMs and retrievers, with both white-box and black-box configurations analyzed. The work provides a rigorous evaluation framework, demonstrates superior execution over baselines, and highlights the need for defense-centric tool-augmentation strategies. It underscores practical risks in real-world deployments and offers guidance for designing robust, secure tool-enabled LLM systems.

Abstract

Tool-calling has changed Large Language Model (LLM) applications by integrating external tools, significantly enhancing their functionality across diverse tasks. However, this integration also introduces new security vulnerabilities, particularly in the tool scheduling mechanisms of LLM, which have not been extensively studied. To fill this gap, we present ToolCommander, a novel framework designed to exploit vulnerabilities in LLM tool-calling systems through adversarial tool injection. Our framework employs a well-designed two-stage attack strategy. Firstly, it injects malicious tools to collect user queries, then dynamically updates the injected tools based on the stolen information to enhance subsequent attacks. These stages enable ToolCommander to execute privacy theft, launch denial-of-service attacks, and even manipulate business competition by triggering unscheduled tool-calling. Notably, the ASR reaches 91.67% for privacy theft and hits 100% for denial-of-service and unscheduled tool calling in certain cases. Our work demonstrates that these vulnerabilities can lead to severe consequences beyond simple misuse of tool-calling systems, underscoring the urgent need for robust defensive strategies to secure LLM Tool-calling systems.

Paper Structure

This paper contains 57 sections, 1 equation, 4 figures, 13 tables.

Table of Contents

  1. Introduction
  2. Tool Platform:
  3. Retriever:
  4. LLM:
  5. Threat Model
  6. Attacker's Objectives
  7. Attacker's Knowledge and Capabilities
  8. Conditions for a Successful Attack
  9. Attack Constraints
  10. ToolCommander Framework
  11. Framework Overview
  12. (PT):
  13. (DoS):
  14. (UTC):
  15. Constructing Tools Satisfying Conditions for Successful Attacks
  16. ...and 42 more sections

Figures (4)

  • Figure 1: Overall Visualization of LLM Tool-Calling System.
  • Figure 2: Overview of ToolCommander Framework. By injecting Manipulator Tools to be retrieved and invoked by Tool-calling systems, we exploit tool responses to manipulate the tool scheduling process, resulting in privacy theft, denial-of-service (DoS), and unscheduled tool-calling attacks.
  • Figure 3: Attack Success Rate for Retrieval and Privacy Theft on the Test Set at Various Injection Percentages, using the ToolBench Retriever and Llama3 LLM.
  • Figure 4: Distribution of the Perplexity value of Black-box Manipulator Tools, White-box Manipulator Tools and Original Normal Tools