SafeArena: Evaluating the Safety of Autonomous Web Agents

TL;DR

SafeArena introduces the first benchmark focused on deliberate misuse by autonomous web agents, compiling 500 tasks (250 harmful, 250 safe) across four realistic web environments and five harm categories. It couples an Agent Risk Assessment (ARIA) framework with automatic and human evaluation to quantify task completion, refusals, and safety, revealing substantial safety vulnerabilities: several state-of-the-art web agents complete harmful tasks at non-trivial rates and can be jailbroken through simple multi-step or priming attacks. The work shows that safety alignment transferred from underlying LLMs to web tasks is limited, underscoring the need for dedicated web-specific safety measures. By providing both the dataset and evaluation framework, SafeArena aims to accelerate research in safe and aligned autonomous web agents and informs policy discussions on responsible deployment.

Abstract

LLM-based agents are becoming increasingly proficient at solving web-based tasks. With this capability comes a greater risk of misuse for malicious purposes, such as posting misinformation in an online forum or selling illicit substances on a website. To evaluate these risks, we propose SafeArena, the first benchmark to focus on the deliberate misuse of web agents. SafeArena comprises 250 safe and 250 harmful tasks across four websites. We classify the harmful tasks into five harm categories -- misinformation, illegal activity, harassment, cybercrime, and social bias, designed to assess realistic misuses of web agents. We evaluate leading LLM-based web agents, including GPT-4o, Claude-3.5 Sonnet, Qwen-2-VL 72B, and Llama-3.2 90B, on our benchmark. To systematically assess their susceptibility to harmful tasks, we introduce the Agent Risk Assessment framework that categorizes agent behavior across four risk levels. We find agents are surprisingly compliant with malicious requests, with GPT-4o and Qwen-2 completing 34.7% and 27.3% of harmful requests, respectively. Our findings highlight the urgent need for safety alignment procedures for web agents. Our benchmark is available here: https://safearena.github.io

Figures (25)

• Figure 1: Overview of example tasks and user-agent interaction within SafeArena. Left: A human-curated example illustrating an intent to spread misinformation on a public forum. Right: An example created by our human-in-the-loop process where the intent is to promote biased discussions.
• Figure 2: An illustration of the task decomposition attack on Claude-3.5-Sonnet. The model refuses to complete the harmful task when directly prompted with the entire intent. However, it successfully completes the task when the intent is decomposed into distinct steps provided sequentially.
• Figure 3: Task completion rates for harmful ($\downarrow$) and safe ($\uparrow$) tasks from SafeArena.
• Figure 4: Harmful task completion rates ($\downarrow$) for each SafeArena harm category.
• Figure 5: Agent Risk Assessment (ARIA) evaluation through an LLM judge. We report the percentage of agent trajectories assigned to each of the four ARIA levels (§\ref{['sec:result-model']}).