Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

DELRec: Distilling Sequential Pattern to Enhance LLMs-based Sequential Recommendation

Haoyi Zhang, Guohao Sun, Jinhu Lu, Guanfeng Liu, Xiu Susie Fang

TL;DR

DELRec addresses the gap of underutilized semantic signals in sequential recommendation by distilling behavioral patterns from conventional SR models into soft prompts for large language models (LLMs). It introduces a two-stage framework: (1) Distill Pattern from Conventional SR Models, comprising Temporal Analysis and Recommendation Pattern Simulating to create informative soft prompts, and (2) LLMs-based Sequential Recommendation, which injects the distilled prompts and applies parameter-efficient fine-tuning (AdaLoRA) to guide the LLM in SR tasks. Empirically, DELRec achieves consistent, statistically significant improvements over conventional SR backbones and several LLM-based baselines across four real-world datasets, while offering favorable memory and latency characteristics and resilience to cold-start scenarios. The work demonstrates that combining distilled, interpretable patterns with LLM world knowledge yields robust, scalable SR performance with practical implications for building context-aware, data-efficient recommender systems.

Abstract

Sequential recommendation (SR) tasks aim to predict users' next interaction by learning their behavior sequence and capturing the connection between users' past interactions and their changing preferences. Conventional SR models often focus solely on capturing sequential patterns within the training data, neglecting the broader context and semantic information embedded in item titles from external sources. This limits their predictive power and adaptability. Large language models (LLMs) have recently shown promise in SR tasks due to their advanced understanding capabilities and strong generalization abilities. Researchers have attempted to enhance LLMs-based recommendation performance by incorporating information from conventional SR models. However, previous approaches have encountered problems such as 1) limited textual information leading to poor recommendation performance, 2) incomplete understanding and utilization of conventional SR model information by LLMs, and 3) excessive complexity and low interpretability of LLMs-based methods. To improve the performance of LLMs-based SR, we propose a novel framework, Distilling Sequential Pattern to Enhance LLMs-based Sequential Recommendation (DELRec), which aims to extract knowledge from conventional SR models and enable LLMs to easily comprehend and utilize the extracted knowledge for more effective SRs. DELRec consists of two main stages: 1) Distill Pattern from Conventional SR Models, focusing on extracting behavioral patterns exhibited by conventional SR models using soft prompts through two well-designed strategies; 2) LLMs-based Sequential Recommendation, aiming to fine-tune LLMs to effectively use the distilled auxiliary information to perform SR tasks. Extensive experimental results conducted on four real datasets validate the effectiveness of the DELRec framework.

DELRec: Distilling Sequential Pattern to Enhance LLMs-based Sequential Recommendation

TL;DR

DELRec addresses the gap of underutilized semantic signals in sequential recommendation by distilling behavioral patterns from conventional SR models into soft prompts for large language models (LLMs). It introduces a two-stage framework: (1) Distill Pattern from Conventional SR Models, comprising Temporal Analysis and Recommendation Pattern Simulating to create informative soft prompts, and (2) LLMs-based Sequential Recommendation, which injects the distilled prompts and applies parameter-efficient fine-tuning (AdaLoRA) to guide the LLM in SR tasks. Empirically, DELRec achieves consistent, statistically significant improvements over conventional SR backbones and several LLM-based baselines across four real-world datasets, while offering favorable memory and latency characteristics and resilience to cold-start scenarios. The work demonstrates that combining distilled, interpretable patterns with LLM world knowledge yields robust, scalable SR performance with practical implications for building context-aware, data-efficient recommender systems.

Abstract

Sequential recommendation (SR) tasks aim to predict users' next interaction by learning their behavior sequence and capturing the connection between users' past interactions and their changing preferences. Conventional SR models often focus solely on capturing sequential patterns within the training data, neglecting the broader context and semantic information embedded in item titles from external sources. This limits their predictive power and adaptability. Large language models (LLMs) have recently shown promise in SR tasks due to their advanced understanding capabilities and strong generalization abilities. Researchers have attempted to enhance LLMs-based recommendation performance by incorporating information from conventional SR models. However, previous approaches have encountered problems such as 1) limited textual information leading to poor recommendation performance, 2) incomplete understanding and utilization of conventional SR model information by LLMs, and 3) excessive complexity and low interpretability of LLMs-based methods. To improve the performance of LLMs-based SR, we propose a novel framework, Distilling Sequential Pattern to Enhance LLMs-based Sequential Recommendation (DELRec), which aims to extract knowledge from conventional SR models and enable LLMs to easily comprehend and utilize the extracted knowledge for more effective SRs. DELRec consists of two main stages: 1) Distill Pattern from Conventional SR Models, focusing on extracting behavioral patterns exhibited by conventional SR models using soft prompts through two well-designed strategies; 2) LLMs-based Sequential Recommendation, aiming to fine-tune LLMs to effectively use the distilled auxiliary information to perform SR tasks. Extensive experimental results conducted on four real datasets validate the effectiveness of the DELRec framework.
Paper Structure (26 sections, 8 equations, 8 figures, 5 tables)

This paper contains 26 sections, 8 equations, 8 figures, 5 tables.

Table of Contents

  1. INTRODUCTION
  2. RELATED WORK
  3. Conventional Sequential Recommendation
  4. LLMs-based Sequential Recommendation
  5. PRELIMINARY
  6. Task Formulation
  7. Soft Prompt
  8. Parameter Efficient Fine-Tuning
  9. METHODOLOGY
  10. Prompt Construction
  11. Distill Pattern from Conventional SR Models
  12. LLMs-based Sequential Recommendation
  13. EXPERIMENTS
  14. Setup
  15. Datasets
  16. ...and 11 more sections

Figures (8)

  • Figure 1: Demonstration of three paradigms of the integration of conventional SR models with LLMs-based SR.
  • Figure 2: Demonstration of the paradigm of DELRec.
  • Figure 3: Illustrating the proposed DELRec. There are three parts in DELRec: Prompt Construction aims to construct prompts for the next two stages to guide LLMs to perform different tasks. Stage 1: Distill Pattern from Conventional SR Models involves using soft prompts to distill SR patterns from conventional SR models. This stage includes two components: Temporal Analysis and Recommendation Pattern Simulating. Temporal Analysis focuses on performing time analysis on conventional SR models and providing similar time knowledge to LLMs; Recommendation Pattern Simulating aims to simulate the recommendation results of conventional SR models, enabling soft prompts to distill similar recommendation knowledge. Stage 2: LLMs-based Sequential Recommendation aims to insert distilled soft prompts into constructed prompt and fine-tune LLMs, enabling them to predict the ground truth.
  • Figure 4: Demonstration of the prompt for Temporal Analysis.
  • Figure 5: Demonstration of the prompt for Recommendation Pattern Simulating.
  • ...and 3 more figures