Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Collaborative Large Language Model for Recommender Systems

Yaochen Zhu, Liang Wu, Qi Guo, Liangjie Hong, Jundong Li

TL;DR

CLLM4Rec presents a novel generative recommender system that tightly couples the traditional ID-based collaborative filtering paradigm with pretrained large language models. It introduces an extended vocabulary with user/item ID tokens, a soft+hard prompting strategy, and mutually-regularized pretraining to align collaborative and content semantics, followed by a recommendation-oriented finetuning that uses a multinomial head to generate multiple hold-out item recommendations efficiently. Empirical results across four public datasets and a LinkedIn job dataset show that CLLM4Rec outperforms strong baselines, highlighting the value of leveraging LLMs for understanding textual user/item content while preserving user/item collaborative signals. The approach demonstrates improved recommendation quality and stable training, though online inference latency remains a consideration for large deployments, suggesting potential hybrid use with lighter down-stream models in production.

Abstract

Recently, there has been growing interest in developing the next-generation recommender systems (RSs) based on pretrained large language models (LLMs). However, the semantic gap between natural language and recommendation tasks is still not well addressed, leading to multiple issues such as spuriously correlated user/item descriptors, ineffective language modeling on user/item data, inefficient recommendations via auto-regression, etc. In this paper, we propose CLLM4Rec, the first generative RS that tightly integrates the LLM paradigm and ID paradigm of RSs, aiming to address the above challenges simultaneously. We first extend the vocabulary of pretrained LLMs with user/item ID tokens to faithfully model user/item collaborative and content semantics. Accordingly, a novel soft+hard prompting strategy is proposed to effectively learn user/item collaborative/content token embeddings via language modeling on RS-specific corpora, where each document is split into a prompt consisting of heterogeneous soft (user/item) tokens and hard (vocab) tokens and a main text consisting of homogeneous item tokens or vocab tokens to facilitate stable and effective language modeling. In addition, a novel mutual regularization strategy is introduced to encourage CLLM4Rec to capture recommendation-related information from noisy user/item content. Finally, we propose a novel recommendation-oriented finetuning strategy for CLLM4Rec, where an item prediction head with multinomial likelihood is added to the pretrained CLLM4Rec backbone to predict hold-out items based on soft+hard prompts established from masked user-item interaction history, where recommendations of multiple items can be generated efficiently without hallucination. Codes are released at https://github.com/yaochenzhu/llm4rec.

Collaborative Large Language Model for Recommender Systems

TL;DR

CLLM4Rec presents a novel generative recommender system that tightly couples the traditional ID-based collaborative filtering paradigm with pretrained large language models. It introduces an extended vocabulary with user/item ID tokens, a soft+hard prompting strategy, and mutually-regularized pretraining to align collaborative and content semantics, followed by a recommendation-oriented finetuning that uses a multinomial head to generate multiple hold-out item recommendations efficiently. Empirical results across four public datasets and a LinkedIn job dataset show that CLLM4Rec outperforms strong baselines, highlighting the value of leveraging LLMs for understanding textual user/item content while preserving user/item collaborative signals. The approach demonstrates improved recommendation quality and stable training, though online inference latency remains a consideration for large deployments, suggesting potential hybrid use with lighter down-stream models in production.

Abstract

Recently, there has been growing interest in developing the next-generation recommender systems (RSs) based on pretrained large language models (LLMs). However, the semantic gap between natural language and recommendation tasks is still not well addressed, leading to multiple issues such as spuriously correlated user/item descriptors, ineffective language modeling on user/item data, inefficient recommendations via auto-regression, etc. In this paper, we propose CLLM4Rec, the first generative RS that tightly integrates the LLM paradigm and ID paradigm of RSs, aiming to address the above challenges simultaneously. We first extend the vocabulary of pretrained LLMs with user/item ID tokens to faithfully model user/item collaborative and content semantics. Accordingly, a novel soft+hard prompting strategy is proposed to effectively learn user/item collaborative/content token embeddings via language modeling on RS-specific corpora, where each document is split into a prompt consisting of heterogeneous soft (user/item) tokens and hard (vocab) tokens and a main text consisting of homogeneous item tokens or vocab tokens to facilitate stable and effective language modeling. In addition, a novel mutual regularization strategy is introduced to encourage CLLM4Rec to capture recommendation-related information from noisy user/item content. Finally, we propose a novel recommendation-oriented finetuning strategy for CLLM4Rec, where an item prediction head with multinomial likelihood is added to the pretrained CLLM4Rec backbone to predict hold-out items based on soft+hard prompts established from masked user-item interaction history, where recommendations of multiple items can be generated efficiently without hallucination. Codes are released at https://github.com/yaochenzhu/llm4rec.
Paper Structure (39 sections, 10 equations, 4 figures, 4 tables)

This paper contains 39 sections, 10 equations, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Large Language Model (LLM) Basics
  4. LLM in Recommender Systems
  5. Methodology
  6. Problem Formulation
  7. Extension of User/Item Tokens
  8. Vocab Expansion
  9. Token Embeddings
  10. CLLM4Rec Base Model
  11. Mutually-Regularized Pretraining
  12. Recommendation-Specific Corpora
  13. Soft$\bm{+}$Hard Prompting
  14. Mutually-Regularization
  15. Stochastic Item Reordering
  16. ...and 24 more sections

Figures (4)

  • Figure 1: Prospectives of developing the next generation of recommender systems based on pretrained LLMs.
  • Figure 2: The overview of the proposed CLLM4Rec in the mutually-regularized pretraining stage. Mutual regularization for item_$\bm{k}$ is omitted for simplicity.
  • Figure 3: Exemplar review data from the Amazon Beauty dataset mcauley2016addressing, where prior knowledge of natural language can help understand item property and user interests.
  • Figure 4: Sensitivity analysis w.r.t. $\lambda_{c}$, which controls the strength of mutual-regularization for CLLM4Rec.