SafePro: Evaluating the Safety of Professional-Level AI Agents

TL;DR

SafePro tackles safety evaluation for professional-level AI agents by introducing a high-complexity, cross-domain dataset and a robust LLM-based evaluation framework. The benchmark reveals substantial safety vulnerabilities across state-of-the-art models when handling professional tasks, identifying a knowledge-application gap in safety judgments. Through mitigation experiments—agent safety prompts, LLM-based safety classification, and guardrails—the work demonstrates meaningful, though incomplete, improvements and highlights the need for domain-specific safety mechanisms. Collectively, SafePro provides a resource and set of insights to drive safer deployment of advanced AI agents in professional settings.

Abstract

Large language model-based agents are rapidly evolving from simple conversational assistants into autonomous systems capable of performing complex, professional-level tasks in various domains. While these advancements promise significant productivity gains, they also introduce critical safety risks that remain under-explored. Existing safety evaluations primarily focus on simple, daily assistance tasks, failing to capture the intricate decision-making processes and potential consequences of misaligned behaviors in professional settings. To address this gap, we introduce \textbf{SafePro}, a comprehensive benchmark designed to evaluate the safety alignment of AI agents performing professional activities. SafePro features a dataset of high-complexity tasks across diverse professional domains with safety risks, developed through a rigorous iterative creation and review process. Our evaluation of state-of-the-art AI models reveals significant safety vulnerabilities and uncovers new unsafe behaviors in professional contexts. We further show that these models exhibit both insufficient safety judgment and weak safety alignment when executing complex professional tasks. In addition, we investigate safety mitigation strategies for improving agent safety in these scenarios and observe encouraging improvements. Together, our findings highlight the urgent need for robust safety mechanisms tailored to the next generation of professional AI agents.

Figures (6)

• Figure 1: Overview of SafePro benchmark. (Left) The SafePro dataset contains safety tests on various professional sectors and occupations, revealing critical safety risks in current AI agents. (Right) State-of-the-art AI models exhibit high unsafe rates in the SafePro benchmark.
• Figure 2: Data creation process for SafePro. We first define a set of requirements for harmful task creation. The data creators then generate harmful professional tasks through two approaches: benign task transformation and new harmful task generation. Each created task undergoes iterative review and revision to ensure quality.
• Figure 3: Example harmful task from SafePro.
• Figure 4: Examples of unsafe actions by AI agents in SafePro benchmark.
• Figure 5: Comparison of unsafe rates (%) with and without safety prompts across three models.