Improving LLMs for Recommendation with Out-Of-Vocabulary Tokens

TL;DR

This work addresses the limitation of in-vocabulary IDs when using LLMs for recommender systems by introducing META ID, a framework that learns OOV tokens from meta-path-based user-item interactions. Meta-path sampling followed by skip-gram embedding yields hierarchical representations that are clustered into coarse and fine-grained OOV tokens, which are then integrated into the LLM via a representation augmentation layer and vocabulary expansion. The authors define memorization score (MS) and diversity score (DS) to quantify the quality of ID representations and show that META ID improves both metrics, leading to superior performance across sequential, direct, rating, explanation, and review tasks on real Amazon datasets. The approach demonstrates robust gains and provides a scalable path to richer user/item representations in LLM-backed recommender systems, with future directions toward cold-start and more complex hierarchical token schemes.

Abstract

Characterizing users and items through vector representations is crucial for various tasks in recommender systems. Recent approaches attempt to apply Large Language Models (LLMs) in recommendation through a question and answer format, where real users and items (e.g., Item No.2024) are represented with in-vocabulary tokens (e.g., "item", "20", "24"). However, since LLMs are typically pretrained on natural language tasks, these in-vocabulary tokens lack the expressive power for distinctive users and items, thereby weakening the recommendation ability even after fine-tuning on recommendation tasks. In this paper, we explore how to effectively tokenize users and items in LLM-based recommender systems. We emphasize the role of out-of-vocabulary (OOV) tokens in addition to the in-vocabulary ones and claim the memorization of OOV tokens that capture correlations of users/items as well as diversity of OOV tokens. By clustering the learned representations from historical user-item interactions, we make the representations of user/item combinations share the same OOV tokens if they have similar properties. Furthermore, integrating these OOV tokens into the LLM's vocabulary allows for better distinction between users and items and enhanced capture of user-item relationships during fine-tuning on downstream tasks. Our proposed framework outperforms existing state-of-the-art methods across various downstream recommendation tasks.

Figures (8)

• Figure 1: (a) Two ID strategies for characterizing items through tokens in LLM-based recommendation. (b) Left: Mapping problems caused by numeric tokens when characterizing items. Two unrelated items might share the same numeric tokens, and consecutive numeric tokens are treated similarly by LLMs. Right: Heatmaps of item representations similarity learned with Random ID (RID), Sequential ID (SID) and META ID on Toys dataset. We randomly sample 50 items and take their ID representations to calculate cosine similarity. The ground Truth is their adjusted cosine similarities in the Equation \ref{['eq:adjusted_sim']}.
• Figure 2: (a) T-SNE visualizations of ID representations on Sports dataset; (b) Comparison of DS and MS of ID representations from RID, SID, CID and META ID.
• Figure 3: Illustration of the proposed framework: (a) The first step involves sampling meta-paths to train a skip-gram model to learn the representation of users and items. Following this, users and items are represented by META IDs constructed of OOV token combinations through clustering their representations. (b) A large language model (LLM) integrated with META ID is utilized for sequential recommendation. The LLM first encodes the input sequence into tokens and then lookup their embeddings through the token embedding layer. Here, the OOV tokens undergo an extra transformation layer before the token embedding layer of LLMs as a representation augmentation.
• Figure 4: Correlation between DS and MS with NDCG@10 on sequential recommendation task.
• Figure 5: Performance comparison of (a) different OOV token size within META ID on three datasets; (b) whether indexing both items and users on Beauty dataset.