From Assistant to Double Agent: Formalizing and Benchmarking Attacks on OpenClaw for Personalized Local AI Agent

TL;DR

This work tackles security of personalized LLM agents by introducing PASB, an end-to-end black-box evaluation framework that models persistent context, long-horizon tool usage, and private asset domains. PASB extends prior benchmarks with realistic personalization, auditable private assets, and memory-based risk propagation across interactive steps, and formalizes attack tasks via a set of primitives and success predicates. Through a case study on OpenClaw, it reveals vulnerabilities across prompt processing, external content, tool usage, and memory retrieval, with attacks capable of leaking private assets, executing unsafe actions, or persisting harmful influence even after injections stop. The framework offers automated, reproducible evaluation and baseline defenses (Delimiter, Sandwich, Instruction Prevention), highlighting that defenses limited to prompt-layer isolation are insufficient for practical, real-world deployments and informing future defenses that address end-to-end toolchains and long-horizon propagation.

Abstract

Although large language model (LLM)-based agents, exemplified by OpenClaw, are increasingly evolving from task-oriented systems into personalized AI assistants for solving complex real-world tasks, their practical deployment also introduces severe security risks. However, existing agent security research and evaluation frameworks primarily focus on synthetic or task-centric settings, and thus fail to accurately capture the attack surface and risk propagation mechanisms of personalized agents in real-world deployments. To address this gap, we propose Personalized Agent Security Bench (PASB), an end-to-end security evaluation framework tailored for real-world personalized agents. Building upon existing agent attack paradigms, PASB incorporates personalized usage scenarios, realistic toolchains, and long-horizon interactions, enabling black-box, end-to-end security evaluation on real systems. Using OpenClaw as a representative case study, we systematically evaluate its security across multiple personalized scenarios, tool capabilities, and attack types. Our results indicate that OpenClaw exhibits critical vulnerabilities at different execution stages, including user prompt processing, tool usage, and memory retrieval, highlighting substantial security risks in personalized agent deployments. The code for the proposed PASB framework is available at https://github.com/AstorYH/PASB.

Figures (2)

• Figure 1: Threat landscape of the Personalized LLM Agent. The agent interacts with an External Environment (content hubs and tools) and maintains a Private Memory.
• Figure 2: Example of Indirect Prompt Injection Attack on Openclaw.