Risky-Bench: Probing Agentic Safety Risks under Real-World Deployment

TL;DR

This work addresses safety risks for LLM-based agents operating in real-world settings where harms extend beyond text to actions and tool use. It introduces Risky-Bench, a deployment-grounded evaluation framework that defines a domain-agnostic safety risk space, derives context-aware rubrics, and probes risks under varying threat models through realistic task execution. Applied to life-assist tasks, Risky-Bench reveals substantial safety vulnerabilities across multiple agent families, with attack success rates varying widely and particular weaknesses in memory, tool feedback, and system prompt channels. The framework is designed to extend to other deployment contexts, enabling environment-specific safety benchmarks and informing safer agent design.

Abstract

Large Language Models (LLMs) are increasingly deployed as agents that operate in real-world environments, introducing safety risks beyond linguistic harm. Existing agent safety evaluations rely on risk-oriented tasks tailored to specific agent settings, resulting in limited coverage of safety risk space and failing to assess agent safety behavior during long-horizon, interactive task execution in complex real-world deployments. Moreover, their specialization to particular agent settings limits adaptability across diverse agent configurations. To address these limitations, we propose Risky-Bench, a framework that enables systematic agent safety evaluation grounded in real-world deployment. Risky-Bench organizes evaluation around domain-agnostic safety principles to derive context-aware safety rubrics that delineate safety space, and systematically evaluates safety risks across this space through realistic task execution under varying threat assumptions. When applied to life-assist agent settings, Risky-Bench uncovers substantial safety risks in state-of-the-art agents under realistic execution conditions. Moreover, as a well-structured evaluation pipeline, Risky-Bench is not confined to life-assist scenarios and can be adapted to other deployment settings to construct environment-specific safety evaluations, providing an extensible methodology for agent safety assessment.

Figures (7)

• Figure 1: Example of a realistic safety risk arising in a delivery scenario. Despite recognizing user’s allergy constraint, the agent is misled by adversarially injected product metadata and orders an unsafe product for the user, resulting in a health-related safety risk.
• Figure 2: Overview of the Risky-Bench. The framework proceeds in three stages: (1) defining the safety space of life agent via instantiation of safety principles into operational rubrics under its deployment settings (Sec \ref{['sec:risk_space']}); (2) Probing safety risks of life agent under the defined safety space during real-world task execution, conditioned on varying threat model assumptions (Sec \ref{['sec:risk_probing']}); (3) Evaluating life agent trajectories with an LLM-as-judge and human-in-the-loop verification, guided by the defined safety rubrics (Section \ref{['sec:evaluation']}).
• Figure 3: Overview of Risky-Bench's safety rubric construction process. Foundational safety principles are conditioned on deployment scenarios and agent capabilities, and instantiated into concrete, operational safety rubrics.
• Figure 4: Overview of agent attack surfaces across different levels of adversarial access. Based on the key components of LLM-based agents, we identify five distinct attack surfaces under three adversarial capability assumptions, corresponding to black-box, grey-box, and white-box access.
• Figure 5: Attack success rates (ASR) for seven agents across three domains under safety risks from five attack surfaces, defined by increasing threat model assumptions as illustrated in Sec \ref{['sec:risk_probing']} and Figure \ref{['fig:threat_model']}. Darker colors indicate higher ASR and, consequently, more severe safety risks.