PeaPOD: Personalized Prompt Distillation for Generative Recommendation

TL;DR

PeaPOD addresses the challenge of learning personalized signals in LLM-based recommender systems by distilling user and collaborative preferences into personalized soft prompts. It constructs these prompts through a compositional attention mechanism over the top-$n$ similar users, yielding a context vector $\boldsymbol{p}$ used across sequential, top-n, and explanation tasks. Empirical results on three Amazon domains show PeaPOD achieving state-of-the-art or competitive performance, with benefits arising from combining individual and group-level signals and from multi-head attention that captures diverse user relationships. The work demonstrates the practical potential of collaborative soft prompts for scalable, explainable recommendations, while acknowledging limitations like the exclusion of metadata and fairness considerations that warrant future work.

Abstract

Recently, researchers have investigated the capabilities of Large Language Models (LLMs) for generative recommender systems. Existing LLM-based recommender models are trained by adding user and item IDs to a discrete prompt template. However, the disconnect between IDs and natural language makes it difficult for the LLM to learn the relationship between users. To address this issue, we propose a PErsonAlized PrOmpt Distillation (PeaPOD) approach, to distill user preferences as personalized soft prompts. Considering the complexities of user preferences in the real world, we maintain a shared set of learnable prompts that are dynamically weighted based on the user's interests to construct the user-personalized prompt in a compositional manner. Experimental results on three real-world datasets demonstrate the effectiveness of our PeaPOD model on sequential recommendation, top-n recommendation, and explanation generation tasks.

Figures (5)

• Figure 1: Comparison of soft prompt architectures for multi-task generative recommendation. Unlike previous methods, PeaPOD utilizes the query-key-value mechanism from attention to generate a collaborative user prompt that captures group-level preferences based on the target user's individual preferences.
• Figure 2: Full PeaPOD model architecture. We first initialize a set of user embeddings using probabilistic matrix factorization. We then apply multi-head attention, where the user embedding of target user $u$ is passed to the query vector and the embeddings of the top-n most similar users $S_u$ are passed to the key and value vectors. The final output is the collaborative user prompt $\boldsymbol{p}_{mha}$, which dynamically captures collaborative knowledge from $S_u$ based on $u$'s personalized preferences.
• Figure 3: Comparison between PeaPOD-PMF and PeaPOD-BiVAE.
• Figure 4: Comparison between PeaPOD-PMF and an ablated version, PeaPOD (no task), trained without task-specific prompts.
• Figure 5: The effects of different number of heads used for attention on the Beauty dataset. Note that '0' represents multi-layer perceptron (MLP) rather than the number of heads.