Optimizing User Profiles via Contextual Bandits for Retrieval-Augmented LLM Personalization

TL;DR

This work reframes LLM personalization as a contextual bandit problem over user history profiles. By using a Plackett–Luce ranking model to generate ordered K-record profiles and training with a dense log-likelihood reward from a frozen LLM, PURPLE directly optimizes utility of personalized responses rather than mere semantic relevance. Across nine real-world tasks and multiple LLM scales, PURPLE consistently outperforms heuristic baselines and single-record rerankers while maintaining high efficiency. The approach demonstrates robust improvements for classification, regression, and generation, and is especially effective when inter-record dependencies and profile ordering are modeled. This provides a scalable, principled pathway for deploying retrieval-augmented personalization at scale.

Abstract

Large Language Models (LLMs) excel at general-purpose tasks, yet adapting their responses to individual users remains challenging. Retrieval augmentation provides a lightweight alternative to fine-tuning by conditioning LLMs on user history records, and existing approaches typically select these records based on semantic relevance. We argue that relevance serves as an unreliable proxy for utility: a record may be semantically similar to a query yet fail to improve generation quality or even degrade it due to redundancy or conflicting information. To bridge this gap, we propose PURPLE, a contextual bandit framework that oPtimizes UseR Profiles for Llm pErsonalization. In contrast to a greedy selection of the most relevant records, PURPLE treats profile construction as a set generation process and utilizes a Plackett-Luce ranking model to capture complex inter-record dependencies. By training with dense feedback provided by the likelihood of the reference response, our method aligns retrieval directly with generation quality. Extensive experiments on nine personalization tasks demonstrate that PURPLE consistently outperforms strong heuristic and retrieval-augmented baselines in both effectiveness and efficiency, establishing a principled and scalable solution for optimizing user profiles.

Figures (5)

• Figure 1: An illustration of the discrepancy between relevance and utility for a user's past interaction records given a query. The thought bubbles illustrate the potential movie genres that satisfy the underlying user intent for each query.
• Figure 2: Workflow of the proposed PURPLE framework. User records encoder takes a user query and a list of user history records as input, outputting the propensity scores of all records. During training, a Plackett-Luce model is employed to convert the propensity scores into a probability distribution over all possible profiles, followed by sampling $M$ profiles for policy gradient estimation. During inference, records with the top-$K$ propensity scores are provided to the LLM along with the user query to generate a personalized response.
• Figure 3: Performance--throughput tradeoff on News.
• Figure 4: Ranking accuracy across the six LaMP tasks.
• Figure 5: Performance--throughput tradeoff across the six LaMP tasks.