Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Large Language Model Architectures for Sequential Recommendations

Hanbing Wang, Xiaorui Liu, Wenqi Fan, Xiangyu Zhao, Venkataramana Kini, Devendra Yadav, Fei Wang, Zhen Wen, Jiliang Tang, Hui Liu

TL;DR

The paper tackles the inefficiency of LLM-based sequential recommendations, pinpointing beam search and long input lengths as major bottlenecks. It proposes Lite-LLM4Rec, a hierarchical LLM framework that encodes long item context into compact embeddings (Item LLM) and processes sequences with a specialized Recommendation LLM, replacing decoder-based decoding with a simple item projection head. This design yields substantial gains in both accuracy and inference speed on three real-world datasets, outperforming strong LLM baselines by large margins (e.g., Recall@10 improvements up to 46.8% and NDCG@20 improvements up to 51.4% on ML-1m) and achieving near-100% efficiency improvements over prior methods. The work also validates the robustness of the approach through ablations and cross-backbone experiments, and it provides practical insights into reducing input length and avoiding beam search for sequential recommendation tasks, with plans for open-sourcing the implementation.

Abstract

Recently, sequential recommendation has been adapted to the LLM paradigm to enjoy the power of LLMs. LLM-based methods usually formulate recommendation information into natural language and the model is trained to predict the next item in an auto-regressive manner. Despite their notable success, the substantial computational overhead of inference poses a significant obstacle to their real-world applicability. In this work, we endeavor to streamline existing LLM-based recommendation models and propose a simple yet highly effective model Lite-LLM4Rec. The primary goal of Lite-LLM4Rec is to achieve efficient inference for the sequential recommendation task. Lite-LLM4Rec circumvents the beam search decoding by using a straight item projection head for ranking scores generation. This design stems from our empirical observation that beam search decoding is ultimately unnecessary for sequential recommendations. Additionally, Lite-LLM4Rec introduces a hierarchical LLM structure tailored to efficiently handle the extensive contextual information associated with items, thereby reducing computational overhead while enjoying the capabilities of LLMs. Experiments on three publicly available datasets corroborate the effectiveness of Lite-LLM4Rec in both performance and inference efficiency (notably 46.8% performance improvement and 97.28% efficiency improvement on ML-1m) over existing LLM-based methods. Our implementations will be open sourced.

Rethinking Large Language Model Architectures for Sequential Recommendations

TL;DR

The paper tackles the inefficiency of LLM-based sequential recommendations, pinpointing beam search and long input lengths as major bottlenecks. It proposes Lite-LLM4Rec, a hierarchical LLM framework that encodes long item context into compact embeddings (Item LLM) and processes sequences with a specialized Recommendation LLM, replacing decoder-based decoding with a simple item projection head. This design yields substantial gains in both accuracy and inference speed on three real-world datasets, outperforming strong LLM baselines by large margins (e.g., Recall@10 improvements up to 46.8% and NDCG@20 improvements up to 51.4% on ML-1m) and achieving near-100% efficiency improvements over prior methods. The work also validates the robustness of the approach through ablations and cross-backbone experiments, and it provides practical insights into reducing input length and avoiding beam search for sequential recommendation tasks, with plans for open-sourcing the implementation.

Abstract

Recently, sequential recommendation has been adapted to the LLM paradigm to enjoy the power of LLMs. LLM-based methods usually formulate recommendation information into natural language and the model is trained to predict the next item in an auto-regressive manner. Despite their notable success, the substantial computational overhead of inference poses a significant obstacle to their real-world applicability. In this work, we endeavor to streamline existing LLM-based recommendation models and propose a simple yet highly effective model Lite-LLM4Rec. The primary goal of Lite-LLM4Rec is to achieve efficient inference for the sequential recommendation task. Lite-LLM4Rec circumvents the beam search decoding by using a straight item projection head for ranking scores generation. This design stems from our empirical observation that beam search decoding is ultimately unnecessary for sequential recommendations. Additionally, Lite-LLM4Rec introduces a hierarchical LLM structure tailored to efficiently handle the extensive contextual information associated with items, thereby reducing computational overhead while enjoying the capabilities of LLMs. Experiments on three publicly available datasets corroborate the effectiveness of Lite-LLM4Rec in both performance and inference efficiency (notably 46.8% performance improvement and 97.28% efficiency improvement on ML-1m) over existing LLM-based methods. Our implementations will be open sourced.
Paper Structure (27 sections, 6 equations, 5 figures, 7 tables, 1 algorithm)

This paper contains 27 sections, 6 equations, 5 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. Sequential Recommendation
  4. LLM-based Recommendation
  5. Preliminaries
  6. Problem Statement and Notations
  7. Efficiency Analysis
  8. Effectiveness Analysis
  9. Discussion
  10. The Proposed Framework
  11. An Overview
  12. Hierarchical LLM Structure
  13. Item Projection Head
  14. Model Training
  15. Experiment
  16. ...and 12 more sections

Figures (5)

  • Figure 1: An Illustration of LLM-based sequential recommendations.
  • Figure 2: Inference time of different components for a batch of 32 users. 'Beam_search' refers to the beam search decoding process, 'Decoding' transfers the latent representation to token IDs, and 'Others' includes data preparation, metrics calculations etc.
  • Figure 3: An overview of the architecture of Lite-LLM4Rec.
  • Figure 4: Ablation study results of R@10, R@20, N@10 and N@20 on ML-1m and Toys dataset.
  • Figure 5: Results of Top-N recommendation on ML-1m.