Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Leveraging Large Language Models for Pre-trained Recommender Systems

Zhixuan Chu, Hongyan Hao, Xin Ouyang, Simeng Wang, Yan Wang, Yue Shen, Jinjie Gu, Qing Cui, Longfei Li, Siqiao Xue, James Y Zhang, Sheng Li

TL;DR

This work presents RecSysLLM, a pre-trained recommender model built on large language models that preserves LLM reasoning while injecting recommendation-domain knowledge through a three-phase design (data, training, inference). It introduces entity-aware masking, span order, and dynamic positional encoding to fuse semantic text understanding with user-item interactions, demonstrated across rating, sequential, explanation, review, and direct tasks on Amazon datasets and a real-world Alipay dataset. The results show strong zero-shot generalization and competitive performance with strong baselines, highlighting the potential for unified, scalable recommendation systems that leverage pre-trained language models. The approach offers practical advantages in cross-task adaptability and efficient fine-tuning via LoRA, with implications for broader multimodal and knowledge-infused recommender applications.

Abstract

Recent advancements in recommendation systems have shifted towards more comprehensive and personalized recommendations by utilizing large language models (LLM). However, effectively integrating LLM's commonsense knowledge and reasoning abilities into recommendation systems remains a challenging problem. In this paper, we propose RecSysLLM, a novel pre-trained recommendation model based on LLMs. RecSysLLM retains LLM reasoning and knowledge while integrating recommendation domain knowledge through unique designs of data, training, and inference. This allows RecSysLLM to leverage LLMs' capabilities for recommendation tasks in an efficient, unified framework. We demonstrate the effectiveness of RecSysLLM on benchmarks and real-world scenarios. RecSysLLM provides a promising approach to developing unified recommendation systems by fully exploiting the power of pre-trained language models.

Leveraging Large Language Models for Pre-trained Recommender Systems

TL;DR

This work presents RecSysLLM, a pre-trained recommender model built on large language models that preserves LLM reasoning while injecting recommendation-domain knowledge through a three-phase design (data, training, inference). It introduces entity-aware masking, span order, and dynamic positional encoding to fuse semantic text understanding with user-item interactions, demonstrated across rating, sequential, explanation, review, and direct tasks on Amazon datasets and a real-world Alipay dataset. The results show strong zero-shot generalization and competitive performance with strong baselines, highlighting the potential for unified, scalable recommendation systems that leverage pre-trained language models. The approach offers practical advantages in cross-task adaptability and efficient fine-tuning via LoRA, with implications for broader multimodal and knowledge-infused recommender applications.

Abstract

Recent advancements in recommendation systems have shifted towards more comprehensive and personalized recommendations by utilizing large language models (LLM). However, effectively integrating LLM's commonsense knowledge and reasoning abilities into recommendation systems remains a challenging problem. In this paper, we propose RecSysLLM, a novel pre-trained recommendation model based on LLMs. RecSysLLM retains LLM reasoning and knowledge while integrating recommendation domain knowledge through unique designs of data, training, and inference. This allows RecSysLLM to leverage LLMs' capabilities for recommendation tasks in an efficient, unified framework. We demonstrate the effectiveness of RecSysLLM on benchmarks and real-world scenarios. RecSysLLM provides a promising approach to developing unified recommendation systems by fully exploiting the power of pre-trained language models.
Paper Structure (29 sections, 1 equation, 4 figures, 10 tables)

This paper contains 29 sections, 1 equation, 4 figures, 10 tables.

Table of Contents

  1. Introduction
  2. RecSysLLM Pretraining Mechanism
  3. Data Phase
  4. Training Phase
  5. Mask Mechanism.
  6. Span Order.
  7. Positional Encoding.
  8. Inference phase
  9. Experiments
  10. Experimental Setup
  11. Datasets.
  12. Metrics.
  13. Baselines for Multiple Tasks
  14. Implementation
  15. Performance.
  16. ...and 14 more sections

Figures (4)

  • Figure 1: This is the framework of RecSysLLM based on a pre-trained generative language model (GLM). To transform the GLM into a specialized model for recommendation systems, several modifications are made while preserving the core knowledge and capabilities of the original language model architecture, such as the new mask mechanism, span order, positional encoding, dynamic position mechanism, and so on.
  • Figure 2: This is the dynamic position mechanism. When one token is generated, it will be judged as one part of an entity or not. If it and the previous token belong to one entity, the intra-position id will continue to grow. Otherwise, it will start at $1$ again.
  • Figure 3: The HR@1 with different rank $r$ of LoRA.
  • Figure 4: The case studies of ChatGPT and GPT-4 for next item recommendation in the real-world dataset.