Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Evaluating Privilege Usage of Agents on Real-World Tools

Quan Zhang, Lianhang Fu, Lvsi Lian, Gwihwan Go, Yujue Wang, Chijin Zhou, Yu Jiang, Geguang Pu

Abstract

Equipping LLM agents with real-world tools can substantially improve productivity. However, granting agents autonomy over tool use also transfers the associated privileges to both the agent and the underlying LLM. Improper privilege usage may lead to serious consequences, including information leakage and infrastructure damage. While several benchmarks have been built to study agents' security, they often rely on pre-coded tools and restricted interaction patterns. Such crafted environments differ substantially from the real-world, making it hard to assess agents' security capabilities in critical privilege control and usage. Therefore, we propose GrantBox, a security evaluation sandbox for analyzing agent privilege usage. GrantBox automatically integrates real-world tools and allows LLM agents to invoke genuine privileges, enabling the evaluation of privilege usage under prompt injection attacks. Our results indicate that while LLMs exhibit basic security awareness and can block some direct attacks, they remain vulnerable to more sophisticated attacks, resulting in an average attack success rate of 84.80% in carefully crafted scenarios.

Evaluating Privilege Usage of Agents on Real-World Tools

Abstract

Equipping LLM agents with real-world tools can substantially improve productivity. However, granting agents autonomy over tool use also transfers the associated privileges to both the agent and the underlying LLM. Improper privilege usage may lead to serious consequences, including information leakage and infrastructure damage. While several benchmarks have been built to study agents' security, they often rely on pre-coded tools and restricted interaction patterns. Such crafted environments differ substantially from the real-world, making it hard to assess agents' security capabilities in critical privilege control and usage. Therefore, we propose GrantBox, a security evaluation sandbox for analyzing agent privilege usage. GrantBox automatically integrates real-world tools and allows LLM agents to invoke genuine privileges, enabling the evaluation of privilege usage under prompt injection attacks. Our results indicate that while LLMs exhibit basic security awareness and can block some direct attacks, they remain vulnerable to more sophisticated attacks, resulting in an average attack success rate of 84.80% in carefully crafted scenarios.

Paper Structure

This paper contains 11 sections, 2 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Method
  3. MCP Server Manager
  4. Request Generator
  5. MCP Server Sandbox
  6. Experiment
  7. Evaluation Setup
  8. Requests Analysis
  9. LLM Security Evaluation
  10. Discussion
  11. Conclusion

Figures (2)

  • Figure 1: Overview of GrantBox Framework. It utilizes an MCP server manager to support seamless deployment and monitoring of MCP servers. Moreover, based on the available servers, a request generator is designed to create diverse benign and adversarial requests. Lastly, an isolated MCP server sandbox ensures safe execution and quick restoration of environments.
  • Figure 2: Diversity Analysis of Generated Requests.