When Personalization Legitimizes Risks: Uncovering Safety Vulnerabilities in Personalized Dialogue Agents

TL;DR

The paper identifies intent legitimation, a safety failure in memory-augmented personalized dialogue agents where benign personal memories bias intent inference. It introduces PS-Bench to systematically evaluate safety under long-term personalization, including Thematic Chat History Augmentation and Persona-Grounded Harmful Queries, across multiple LLM backbones and memory frameworks. Empirical results show that personalization degrades safety in a memory-design–dependent manner, with evidence from internal representations that retrieved memories shift harmful queries toward perceived legitimacy. A lightweight detection-reflection intervention mitigates safety degradation while largely preserving personalization utility, underscoring the need for safety-aware evaluation and mitigation in long-term personalized systems.

Abstract

Long-term memory enables large language model (LLM) agents to support personalized and sustained interactions. However, most work on personalized agents prioritizes utility and user experience, treating memory as a neutral component and largely overlooking its safety implications. In this paper, we reveal intent legitimation, a previously underexplored safety failure in personalized agents, where benign personal memories bias intent inference and cause models to legitimize inherently harmful queries. To study this phenomenon, we introduce PS-Bench, a benchmark designed to identify and quantify intent legitimation in personalized interactions. Across multiple memory-augmented agent frameworks and base LLMs, personalization increases attack success rates by 15.8%-243.7% relative to stateless baselines. We further provide mechanistic evidence for intent legitimation from internal representations space, and propose a lightweight detection-reflection method that effectively reduces safety degradation. Overall, our work provides the first systematic exploration and evaluation of intent legitimation as a safety failure mode that naturally arises from benign, real-world personalization, highlighting the importance of assessing safety under long-term personal context. WARNING: This paper may contain harmful content.

Figures (17)

• Figure 1: The dual-edged role of memory in personalized agents. Left: Memory augmentation enables personalization by incorporating user-specific preferences, yielding more contextualized responses than a stateless agent. Right: On AdvBench, memory retrieval increases the attack success rate (ASR): a stateless agent refuses a malicious request, while a personalized agent retrieves related memories and reframes the request as acceptable, a phenomenon we term intent legitimization, driven by semantic overgeneralization from personal context.
• Figure 2: Overview of PS-Bench for evaluating safety under personalization. (a) Base setup of a memory-augmented agent evaluated on harmful queries. (b) Thematic chat history augmentation that adds sustained, benign life-theme signals to user memory through synthesized dialogues. (c) Persona-grounded harmful queries that express unsafe intents in a user-natural and persona-consistent manner based on role profiles and personal events.
• Figure 3: Heatmap of $\Delta$ASR for GPT-4o within the LDAgent framework under Thematic Chat History Augmentation relative to the stateless baseline. Rows denote augmented themes, and columns correspond to harmful query categories.
• Figure 4: Results on PS-Bench-Hard across roles with Qwen3-235B-A22B. Bars and curves represent MemOS and the stateless baseline, respectively. Horizontal dashed lines indicate their corresponding performance on the base setting of PS-Bench for reference.
• Figure 5: PCA visualization of representation shifts in Qwen3-8B. Top: Marginal density distributions along the first principal component.