Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety

Sanidhya Vijayvargiya, Aditya Bharat Soni, Xuhui Zhou, Zora Zhiruo Wang, Nouha Dziri, Graham Neubig, Maarten Sap

TL;DR

OpenAgentSafety is introduced, a comprehensive and modular framework for evaluating agent behavior across eight critical risk categories that combines rule-based analysis with LLM-as-judge assessments to detect both overt and subtle unsafe behaviors.

Abstract

Recent advances in AI agents capable of solving complex, everyday tasks, from scheduling to customer service, have enabled deployment in real-world settings, but their possibilities for unsafe behavior demands rigorous evaluation. While prior benchmarks have attempted to assess agent safety, most fall short by relying on simulated environments, narrow task domains, or unrealistic tool abstractions. We introduce OpenAgentSafety, a comprehensive and modular framework for evaluating agent behavior across eight critical risk categories. Unlike prior work, our framework evaluates agents that interact with real tools, including web browsers, code execution environments, file systems, bash shells, and messaging platforms; and supports over 350 multi-turn, multi-user tasks spanning both benign and adversarial user intents. OpenAgentSafety is designed for extensibility, allowing researchers to add tools, tasks, websites, and adversarial strategies with minimal effort. It combines rule-based analysis with LLM-as-judge assessments to detect both overt and subtle unsafe behaviors. Empirical analysis of five prominent LLMs in agentic scenarios reveals unsafe behavior in 51.2% of safety-vulnerable tasks with Claude-Sonnet-3.7, to 72.7% with o3-mini, highlighting critical safety vulnerabilities and the need for stronger safeguards before real-world deployment.

OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safety

TL;DR

OpenAgentSafety is introduced, a comprehensive and modular framework for evaluating agent behavior across eight critical risk categories that combines rule-based analysis with LLM-as-judge assessments to detect both overt and subtle unsafe behaviors.

Abstract

Recent advances in AI agents capable of solving complex, everyday tasks, from scheduling to customer service, have enabled deployment in real-world settings, but their possibilities for unsafe behavior demands rigorous evaluation. While prior benchmarks have attempted to assess agent safety, most fall short by relying on simulated environments, narrow task domains, or unrealistic tool abstractions. We introduce OpenAgentSafety, a comprehensive and modular framework for evaluating agent behavior across eight critical risk categories. Unlike prior work, our framework evaluates agents that interact with real tools, including web browsers, code execution environments, file systems, bash shells, and messaging platforms; and supports over 350 multi-turn, multi-user tasks spanning both benign and adversarial user intents. OpenAgentSafety is designed for extensibility, allowing researchers to add tools, tasks, websites, and adversarial strategies with minimal effort. It combines rule-based analysis with LLM-as-judge assessments to detect both overt and subtle unsafe behaviors. Empirical analysis of five prominent LLMs in agentic scenarios reveals unsafe behavior in 51.2% of safety-vulnerable tasks with Claude-Sonnet-3.7, to 72.7% with o3-mini, highlighting critical safety vulnerabilities and the need for stronger safeguards before real-world deployment.

Paper Structure

This paper contains 24 sections, 10 figures, 6 tables.

Table of Contents

  1. Introduction
  2. OpenAgentSafety Framework
  3. Infrastructure For Agent and Environment
  4. Safety Taxonomy and Task Design
  5. Evaluation Approach
  6. Experiments and Results
  7. Experimental Setup
  8. Results
  9. Analysis
  10. RQ1: How does user intent influence unsafe behavior?
  11. RQ2: Which risk categories reveal the most serious vulnerabilities?
  12. RQ3: Do tools enable or restrain unsafe actions?
  13. RQ4: How reliable are LLM judges in detecting unsafe agentic behavior?
  14. Related work
  15. Conclusion, Limitations, and Future Work
  16. ...and 9 more sections

Figures (10)

  • Figure 1: An overview of the OpenAgentSafety framework.
  • Figure 2: Unsafe agent behaviour rates of various LLMs measured using the OpenAgentSafety framework when navigating conflicting user and NPC instructions.
  • Figure 3: Unsafe behaviour rates (%) across user and NPC intents.
  • Figure 4: Unsafe behavior rates (%) across risk categories.
  • Figure 5: Unsafe behavior rates (%) by tools across LLMs.
  • ...and 5 more figures