Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

BenchPreS: A Benchmark for Context-Aware Personalized Preference Selectivity of Persistent-Memory LLMs

Sangyeon Yoon, Sunkyoung Kim, Hyesoo Hong, Wonje Jeung, Yongil Kim, Wooseok Seo, Heuiyeen Yeen, Albert No

Abstract

Large language models (LLMs) increasingly store user preferences in persistent memory to support personalization across interactions. However, in third-party communication settings governed by social and institutional norms, some user preferences may be inappropriate to apply. We introduce BenchPreS, which evaluates whether memory-based user preferences are appropriately applied or suppressed across communication contexts. Using two complementary metrics, Misapplication Rate (MR) and Appropriate Application Rate (AAR), we find even frontier LLMs struggle to apply preferences in a context-sensitive manner. Models with stronger preference adherence exhibit higher rates of over-application, and neither reasoning capability nor prompt-based defenses fully resolve this issue. These results suggest current LLMs treat personalized preferences as globally enforceable rules rather than as context-dependent normative signals.

BenchPreS: A Benchmark for Context-Aware Personalized Preference Selectivity of Persistent-Memory LLMs

Abstract

Large language models (LLMs) increasingly store user preferences in persistent memory to support personalization across interactions. However, in third-party communication settings governed by social and institutional norms, some user preferences may be inappropriate to apply. We introduce BenchPreS, which evaluates whether memory-based user preferences are appropriately applied or suppressed across communication contexts. Using two complementary metrics, Misapplication Rate (MR) and Appropriate Application Rate (AAR), we find even frontier LLMs struggle to apply preferences in a context-sensitive manner. Models with stronger preference adherence exhibit higher rates of over-application, and neither reasoning capability nor prompt-based defenses fully resolve this issue. These results suggest current LLMs treat personalized preferences as globally enforceable rules rather than as context-dependent normative signals.
Paper Structure (31 sections, 2 equations, 13 figures, 6 tables)

This paper contains 31 sections, 2 equations, 13 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Persistent Memory Systems in LLMs.
  4. Personalization and Preference Following.
  5. BenchPreS: Context-Aware Preference Selectivity in Persistent-Memory LLMs
  6. Problem Formulation
  7. Data Construction
  8. Contexts.
  9. User Profiles.
  10. Gold Labeling.
  11. Evaluation Protocols
  12. Experiments
  13. Experimental Setup
  14. Main Results
  15. Qualitative Examples
  16. ...and 16 more sections

Figures (13)

  • Figure 1: Preference selectivity across models. Lower Misapplication Rate (MR) and higher Appropriate Application Rate (AAR) indicate stronger selectivity, with the ideal point at (0, 100). Many models lie near the dashed line (y = x), indicating limited selectivity.
  • Figure 2: BenchPreS setup overview. Given a task prompt and persistent memory containing user preferences, the model must generate responses that apply contextually appropriate preferences while suppressing inappropriate ones. The top example succeeds, whereas the bottom example fails.
  • Figure 3: Qualitative Failure Cases in Formal Communication Settings. Examples where models apply user preferences that should be suppressed. Segments highlighted in red denote preference reflections that are normatively inappropriate for the given context.
  • Figure 4: Performance comparison of non-reasoning and reasoning-enabled model variants in terms of Misapplication Rate (MR), Appropriate Application Rate (AAR), and IFBench score.
  • Figure 5: Effect of prompt-based mitigation on MR and AAR for Qwen3 235B A22B 2507. Hatched regions indicate changes from the default setting.
  • ...and 8 more figures