MIP against Agent: Malicious Image Patches Hijacking Multimodal OS Agents

TL;DR

The paper tackles the problem of securing multimodal OS agents that rely on screenshot-driven perception and action APIs. It introduces Malicious Image Patches (MIPs) as adversarial screen regions that, when captured, drive the agent to emit a predefined malicious output and execute harmful actions, independent of textual prompts. The authors formalize the OS-agent architecture, define the attack under realistic constraints, and demonstrate both targeted and universal MIPs that generalize across prompts, screen layouts, parsers, and multiple VLMs, with varying degrees of cross-model transfer. They also discuss practical defenses and emphasize the urgency of robust security measures for OS agents given the potential for widespread impact, including propagation and worm-like spread. The findings underscore a fundamental shift in AI safety for agent platforms and motivate future work on defense mechanisms and secure deployment frameworks.

Abstract

Recent advances in operating system (OS) agents have enabled vision-language models (VLMs) to directly control a user's computer. Unlike conventional VLMs that passively output text, OS agents autonomously perform computer-based tasks in response to a single user prompt. OS agents do so by capturing, parsing, and analysing screenshots and executing low-level actions via application programming interfaces (APIs), such as mouse clicks and keyboard inputs. This direct interaction with the OS significantly raises the stakes, as failures or manipulations can have immediate and tangible consequences. In this work, we uncover a novel attack vector against these OS agents: Malicious Image Patches (MIPs), adversarially perturbed screen regions that, when captured by an OS agent, induce it to perform harmful actions by exploiting specific APIs. For instance, a MIP can be embedded in a desktop wallpaper or shared on social media to cause an OS agent to exfiltrate sensitive user data. We show that MIPs generalise across user prompts and screen configurations, and that they can hijack multiple OS agents even during the execution of benign instructions. These findings expose critical security vulnerabilities in OS agents that have to be carefully addressed before their widespread deployment.

Figures (7)

• Figure 1: Illustrating an attack with Malicious Image Patches (MIPs).(1) An adversary (left) utilises an OS agent to craft a MIP that triggers malicious behaviour when captured. (2) The adversary uploads the MIP to a social media platform. (3) A user (right) uses an OS agent to perform benign tasks. The agent takes screenshots for navigation, thereby capturing the adversary's MIP. (4) Upon processing the MIP, the agent deviates from the benign task and outputs a malicious program instead. (5) The malicious program triggers a series of API calls that exfiltrate sensitive data to the adversary.
• Figure 2: Malicious Image Patches (MIPs) in the Wild.MIPs crafted to hijack an OS agent when captured via screenshot are embedded in a desktop background (left) and a social media post (right), making them difficult to detect and capable of widespread dissemination.
• Figure 3: Desktop Setting. Maximum perturbation of a MIP.
• Figure 4: OS Agent Pipeline. Processing steps of the OS agent's components and the illustration of the adversarial attack to craft MIPs.
• Figure 5: Illustration of an OS Agent’s Screen Parser Output. On the one hand, the parser annotates the screenshot with SOMs by overlaying numbered bounding boxes. On the other hand, it generates a structured text description detailing each SOM.