OP-Bench: Benchmarking Over-Personalization for Memory-Augmented Personalized Conversational Agents

TL;DR

This work proposes \textbf{Self-ReCheck}, a lightweight, model-agnostic memory filtering mechanism that mitigates over-personalization while preserving personalization performance in memory-augmented dialogue systems.

Abstract

Memory-augmented conversational agents enable personalized interactions using long-term user memory and have gained substantial traction. However, existing benchmarks primarily focus on whether agents can recall and apply user information, while overlooking whether such personalization is used appropriately. In fact, agents may overuse personal information, producing responses that feel forced, intrusive, or socially inappropriate to users. We refer to this issue as \emph{over-personalization}. In this work, we formalize over-personalization into three types: Irrelevance, Repetition, and Sycophancy, and introduce \textbf{OP-Bench} a benchmark of 1,700 verified instances constructed from long-horizon dialogue histories. Using \textbf{OP-Bench}, we evaluate multiple large language models and memory-augmentation methods, and find that over-personalization is widespread when memory is introduced. Further analysis reveals that agents tend to retrieve and over-attend to user memories even when unnecessary. To address this issue, we propose \textbf{Self-ReCheck}, a lightweight, model-agnostic memory filtering mechanism that mitigates over-personalization while preserving personalization performance. Our work takes an initial step toward more controllable and appropriate personalization in memory-augmented dialogue systems.

Figures (43)

• Figure 1: Example illustrating three canonical forms of over-personalization. In each card, the top response exhibits over-personalization, while the bottom provides a contextually appropriate reply. From top to bottom: (1) Irrelevance (2) Sycophancy (3) Repetition
• Figure 2: Comparison of over-personalization scores across different base models and memory systems. Lower scores indicate more severe over-personalization. The BASE setting (without memory) consistently achieves the highest scores, while memory mechanisms substantially exacerbate over-personalization across all models.
• Figure 3: Overview of the pipeline for constructing over-personalization evaluation queries. The process consists of three stages: initialization of user profiles and topics, task construction targeting Irrelevance, Sycophancy, and Repetition, and human review for quality control and final query selection.
• Figure 4: Length-normalized attention scores assigned to retrieved memory and user query across all benchmark tasks. Models consistently attend more to memory tokens than to query tokens.
• Figure 5: Embedding-based similarity between user queries and retrieved memories on the Irrelevance task. Memories are retrieved even when semantic relevance is low, especially in fully irrelevant cases.