Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs

TL;DR

PrefEval introduces a long-context, multi-session benchmarking framework to assess LLMs' ability to infer, memorize, and follow user preferences. It combines 3,000 preference-query pairs across 20 topics and three preference forms, evaluated via generation and MCQ tasks with both explicit and implicit preferences. Across 10 state-of-the-art models and up to 100k-token contexts, results show substantial gaps in proactive personalization, with zero-shot accuracy often dropping below 10% within ten turns, and long-context degradation even with advanced prompting and retrieval methods. Finetuning on PrefEval yields significant improvements and better generalization, highlighting PrefEval as a practical resource for advancing personalized conversational agents.

Abstract

Large Language Models (LLMs) are increasingly used as chatbots, yet their ability to personalize responses to user preferences remains limited. We introduce PrefEval, a benchmark for evaluating LLMs' ability to infer, memorize and adhere to user preferences in a long-context conversational setting. PrefEval comprises 3,000 manually curated user preference and query pairs spanning 20 topics. PrefEval contains user personalization or preference information in both explicit and implicit forms, and evaluates LLM performance using a generation and a classification task. With PrefEval, we evaluated the aforementioned preference following capabilities of 10 open-source and proprietary LLMs in multi-session conversations with varying context lengths up to 100k tokens. We benchmark with various prompting, iterative feedback, and retrieval-augmented generation methods. Our benchmarking effort reveals that state-of-the-art LLMs face significant challenges in proactively following users' preferences during conversations. In particular, in zero-shot settings, preference following accuracy falls below 10% at merely 10 turns (~3k tokens) across most evaluated models. Even with advanced prompting and retrieval methods, preference following still deteriorates in long-context conversations. Furthermore, we show that fine-tuning on PrefEval significantly improves performance. We believe PrefEval serves as a valuable resource for measuring, understanding, and enhancing LLMs' preference following abilities, paving the way for personalized conversational agents. Our code and dataset are available at https://prefeval.github.io/.

Figures (25)

• Figure 1: PrefEval setup overview. Key components from left to right: 1) Multi-Session Conversation Setup: PrefEval evaluates LLMs' ability to follow user preferences in multi-session conversation, challenging LLMs to handle preference inference, long-range retrieval, and context-aware preference following simultaneously. 2) Preferences and Queries: User preferences can be expressed through both explicit and implicit forms. Queries are designed such that a non-personalized answer would inadvertently conflict with user preferences, testing the LLM's adherence. 3) Tasks and Evaluations: PrefEval includes generation and classification tasks. Generation tasks are evaluated using an LLM-based evaluator to measure preference following accuracy and analyze error types. Classification tasks enable quicker evaluation through multiple-choice questions (MCQ). The two tasks' performances are highly correlated as shown in Fig \ref{['fig:mcq_correlation']}.
• Figure 2: Distribution of domains and topics within PrefEval, which are commonly encountered during conversations with chatbots where users seek recommendations, suggestions, and advice.
• Figure 3: Zero-shot performance of LLMs with explicit preferences, averaged across 20 topics. The x-axis represents the dialogue length between the user's stated preference and the final query, measured by both the number of tokens in the prompt and the number of conversation turns. All LLMs exhibit a rapid decline in accuracy as the number of turns increases.
• Figure 4: Performance comparison of 5 methods across 6 LLMs with explicit preferences on the generation task. Both Reminder and RAG consistently achieve the highest accuracy across models. Notably, Reminder outperforms more complex techniques such as Self-Critic and CoT.
• Figure 5: Comparison of 3 preference forms for 6 LLMs on the generation task, across varying lengths between the stated preference and query. Note that the user preference is stated in the first turn. Results show that implicit preferences are more challenging to infer than explicit preferences.