UQABench: Evaluating User Embedding for Prompting LLMs in Personalized Question Answering

TL;DR

This work tackles the challenge of personalized prompting of LLMs by compressing user interactions into compact embeddings. It introduces UQABench, a three-stage benchmark (pre-training, fine-tuning, evaluating) with three evaluation tasks—sequence understanding, action prediction, and interest perception—based on a Taobao-derived dataset. Through extensive experiments with multiple encoder models, the authors show embedding-based GRs can match or exceed text-based prompts on several tasks, establish a scaling law with model size and sequence length, and demonstrate significant efficiency gains in token usage. The study provides actionable guidance for deploying personalized LLM prompting and offers open data and code to spur further research in the LLM-era recommender setting.

Abstract

Large language models (LLMs) achieve remarkable success in natural language processing (NLP). In practical scenarios like recommendations, as users increasingly seek personalized experiences, it becomes crucial to incorporate user interaction history into the context of LLMs to enhance personalization. However, from a practical utility perspective, user interactions' extensive length and noise present challenges when used directly as text prompts. A promising solution is to compress and distill interactions into compact embeddings, serving as soft prompts to assist LLMs in generating personalized responses. Although this approach brings efficiency, a critical concern emerges: Can user embeddings adequately capture valuable information and prompt LLMs? To address this concern, we propose \name, a benchmark designed to evaluate the effectiveness of user embeddings in prompting LLMs for personalization. We establish a fair and standardized evaluation process, encompassing pre-training, fine-tuning, and evaluation stages. To thoroughly evaluate user embeddings, we design three dimensions of tasks: sequence understanding, action prediction, and interest perception. These evaluation tasks cover the industry's demands in traditional recommendation tasks, such as improving prediction accuracy, and its aspirations for LLM-based methods, such as accurately understanding user interests and enhancing the user experience. We conduct extensive experiments on various state-of-the-art methods for modeling user embeddings. Additionally, we reveal the scaling laws of leveraging user embeddings to prompt LLMs. The benchmark is available online.

Figures (3)

• Figure 1: A brief comparison of SRs and GRs.
• Figure 2: An overview of the standardized evaluation process of user embeddings on UQABench. The left-bottom part demonstrates the encoder model encodes the historical interactions into user embeddings, which requires pre-training. The left-top part shows that user embeddings act as the soft prompt for LLMs to generate personalized responses, which needs fine-tuning. The right side demonstrates the proposed tasks in the evaluating stage and the corresponding Q&A demo.
• Figure 3: Scaling law in terms of the model size.