POPI: Personalizing LLMs via Optimized Natural Language Preference Inference

TL;DR

POPI addresses user-specific LLM personalization by learning a trainable preference-inference module that converts heterogeneous signals into concise natural-language summaries $z$. These summaries condition a shared generation model, enabling personalized responses without per-user weight updates and achieving substantial context-efficiency gains. The method unifies inference and generation under a summary-augmented objective and employs a two-stage training procedure with reinforcement learning for the inference module, followed by fine-tuning the generator; it also supports plug-and-play transfer to frozen off-the-shelf LLMs. Across four benchmarks, POPI demonstrates improved personalization accuracy and dramatic reductions in context overhead, with transferable summaries yielding consistent gains on diverse external models. The approach offers interpretable, transferable personalization signals and points to future directions in lifelong, retrieval-augmented, and multimodal personalization.

Abstract

Large language models (LLMs) achieve strong benchmark performance, yet user experiences remain inconsistent due to diverse preferences in style, tone, and reasoning mode. Nevertheless, existing alignment techniques such as reinforcement learning from human feedback (RLHF) or Direct Preference Optimization (DPO) largely optimize toward population-level averages and overlook individual variation. Naive personalization strategies like per-user fine-tuning are computationally prohibitive, and in-context approaches that prepend raw user signals often suffer from inefficiency and noise. To address these challenges, we propose POPI, a general framework that introduces a preference inference model to distill heterogeneous user signals into concise natural language summaries. These summaries act as transparent, compact, and transferable personalization representations that condition a shared generation model to produce personalized responses. POPI jointly optimizes both preference inference and personalized generation under a unified objective using reinforcement learning, ensuring summaries maximally encode useful preference information. Extensive experiments across four personalization benchmarks demonstrate that POPI consistently improves personalization accuracy while reducing context overhead by a large margin. Moreover, optimized summaries seamlessly transfer to frozen off-the-shelf LLMs, enabling plug-and-play personalization without weight updates.

Figures (5)

• Figure 1: Overview of POPI usage. Heterogeneous user signals (such as textual personas, few-shot preference pairs, and short interaction history) are provided to a preference inference LLM, which produces a concise natural-language preference summary. Given a user prompt, a generation LLM conditions on this summary to produce a personalized response. Both modules are trained with a unified optimization objective. The learned summaries can also steer frozen off-the-shelf LLMs, enabling plug-and-play personalization without accessing LLM parameters.
• Figure 2: Training framework of POPI. Given raw user signals, the preference inference LLM generates a natural language preference summary. This summary, together with preference-labeled data (prompt, chosen response, rejected response), is used to compute a summary-augmented preference optimization objective, which serves as a unified optimization objective for both models. Training proceeds in two stages: (1) the inference LLM is optimized with reinforcement learning, and (2) the generation LLM is fine-tuned via standard backpropagation and gradient descent. For plug-and-play personalization, the second stage can be omitted, and the inferred summaries can be transferred to adapt external LLMs directly.
• Figure 3: The template used when querying GPT-4o to serve as a judge. The template ensures that the model considers both the user's prompt and the ground-truth user description when comparing candidate outputs.
• Figure 4: Left: templates for the preference inference LLM, which transforms raw user signals (or user signals plus prompt) into natural language summaries. Right: templates for the generation LLM, which integrate user prompts, raw signals, or preference summaries to produce personalized responses. These templates render conditioning variables as natural language.
• Figure 5: Qualitative case study on the ELIX dataset. Each row corresponds to one of the five ground-truth personas in ELIX, all responding to the same user prompt: “How does a thermostat control temperature?” The ground-truth personas are not accessible during training. For each persona, we show an example of the optimized preference summary inferred from user signals, and the corresponding personalized response generated (truncated for brevity). The summaries capture stylistic and content-level preferences at different expertise levels, while the generated responses demonstrate how conditioning on these summaries yields appropriately tailored explanations.