VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agents

TL;DR

VPI-Bench introduces Visual Prompt Injection (VPI) as a realistic threat to multimodal agents with full system access and provides a 306-test benchmark across five platforms to evaluate CUAs and BUAs in dynamic, interactive settings. The study demonstrates pervasive vulnerability across state-of-the-art agents, with system prompts offering limited protection and injection timing influencing outcomes. By detailing a threat model, dataset design, evaluation protocol, and extensive experiments, the work highlights the urgent need for robust, context-aware defenses at both the agent and system levels. The public code and dataset enable reproducible benchmarking and guide future research toward safer, trustworthy AI agents in real-world deployments.

Abstract

Computer-Use Agents (CUAs) with full system access enable powerful task automation but pose significant security and privacy risks due to their ability to manipulate files, access user data, and execute arbitrary commands. While prior work has focused on browser-based agents and HTML-level attacks, the vulnerabilities of CUAs remain underexplored. In this paper, we investigate Visual Prompt Injection (VPI) attacks, where malicious instructions are visually embedded within rendered user interfaces, and examine their impact on both CUAs and Browser-Use Agents (BUAs). We propose VPI-Bench, a benchmark of 306 test cases across five widely used platforms, to evaluate agent robustness under VPI threats. Each test case is a variant of a web platform, designed to be interactive, deployed in a realistic environment, and containing a visually embedded malicious prompt. Our empirical study shows that current CUAs and BUAs can be deceived at rates of up to 51% and 100%, respectively, on certain platforms. The experimental results also indicate that system prompt defenses offer only limited improvements. These findings highlight the need for robust, context-aware defenses to ensure the safe deployment of multimodal AI agents in real-world environments. The code and dataset are available at: https://github.com/cua-framework/agents

Figures (5)

• Figure 1: Overview of the threat model on CUAs. Each sample includes: (1) a benign user prompt describing a normal task; (2) a pseudo-authentic web platform that the agent interacts with, which may be compromised; (3) a visual attack prompt injected into the webpage by an attacker; and (4) an environment setup tailored to the visual attack prompt, enabling accurate monitoring of unauthorized actions such as file deletion or data exfiltration.
• Figure 2: Distribution of samples across three dimensions.
• Figure 3: Distribution of model behaviors across five platforms (Amazon, Booking, BBC, Messenger, and Email) for Sonnet 3.7 (top row) and Sonnet 3.5 (bottom row). Each pie chart illustrates the proportion of actions. The red tone indicates successful attempts, orange represents failure cases, and greenish-blue shades denote unattempted actions.
• Figure 4: Comparison of early and late prompt injection attack outcomes on Messenger and Email platforms using Sonnet 3.5 and Sonnet 3.7 models. Bars are stacked to show the proportion of Success and Attempted Only (i.e., failed attempts), under Early Injection and Late Injection scenarios.
• Figure 5: Comparison of model performance across five platforms (Amazon, Booking, BBC, Messenger, and Email) under two conditions: with and without system prompt defense. Each subplot displays the Success Rate (top) and Attempted Rate (bottom) of four different models: Sonnet-3.7 (Computer-Use), Sonnet-3.7 (Browser-Use), GPT-4o (Browser-Use), and Gemini-2.5 (Browser-Use).