Towards Realistic Personalization: Evaluating Long-Horizon Preference Following in Personalized User-LLM Interactions

TL;DR

Results indicate that LLM performance significantly drops as context length grows and preference expression becomes more implicit, and that generalizing user preference understanding to unseen scenarios poses further challenges.

Abstract

Large Language Models (LLMs) are increasingly serving as personal assistants, where users share complex and diverse preferences over extended interactions. However, assessing how well LLMs can follow these preferences in realistic, long-term situations remains underexplored. This work proposes RealPref, a benchmark for evaluating realistic preference-following in personalized user-LLM interactions. RealPref features 100 user profiles, 1300 personalized preferences, four types of preference expression (ranging from explicit to implicit), and long-horizon interaction histories. It includes three types of test questions (multiple-choice, true-or-false, and open-ended), with detailed rubrics for LLM-as-a-judge evaluation. Results indicate that LLM performance significantly drops as context length grows and preference expression becomes more implicit, and that generalizing user preference understanding to unseen scenarios poses further challenges. RealPref and these findings provide a foundation for future research to develop user-aware LLM assistants that better adapt to individual needs. The code is available at https://github.com/GG14127/RealPref.

Figures (8)

• Figure 1: An example of user-LLM interaction: the conversation consists of several sessions on different topics. The user expresses a food preference in a previous session and then, after several sessions, asks the LLM for food-relevant recommendations.
• Figure 2: Generation Pipeline Overview. Starting from user personas, we construct detailed user profiles and biographies, and generate diverse preferences. Based on profiles and preferences, we build rich conversation sessions to simulate user-LLM interaction dynamics, encompassing preference expressions from explicit to implicit. These sessions are concatenated into a long-horizon context for each user. For each preference, we design generation and classification tasks with corresponding evaluation methods.
• Figure 3: Benchmark Configuration Overview. Preference Expression Type (Direct Statement, Contextualized Mention, Stylistic Expression, and Experience Feedback) and Context Configuration (controls the insertion of random conversations) set the context input. Question Type (Multiple-Choice, True-or-False, and Open-Ended) set the test task and evaluation method. Various Improvement Methods (Reminder, Few-Shot CoT, and RAG) are tested with models. Each factor above is related to a research question (RQ) we will explore in the experiments.
• Figure 4: Model Performance across Question Types with Normal context, Experience Feedback expression, Zero-shot.
• Figure 5: Model Performance across Expression Types with Normal context, Zero-shot. Type 1: Explicit - Direct Statement, Type 2: Explicit - Contextualized Mention, Type 3: Implicit - Stylistic Expression, Type 4: Implicit - Experience Feedback.