Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Grained Patch Training for Efficient LLM-based Recommendation

Jiayi Liao, Ruobing Xie, Sihang Li, Xiang Wang, Xingwu Sun, Zhanhui Kang, Xiangnan He

TL;DR

PatchRec introduces a multi-grained patch training framework for LLM-based recommendations to model long-term user histories within limited context windows. It hierarchically compresses item histories into item patches and session patches, and employs a two-stage training process—Patch Pre-training and Patch Fine-tuning—to teach the model to interpret compressed representations and to assign time-aware significance to interactions. The approach yields substantial gains in recommendation accuracy (e.g., HR@20 improvements up to 32%) while dramatically reducing input-token usage (compression ratios up to multi-fold), outperforming TALLRec and other baselines on MovieLens and Goodreads datasets. These results demonstrate that PatchRec enables practical, long-horizon LLM-based sequential recommendations with improved efficiency and robustness for real-world applications.

Abstract

Large Language Models (LLMs) have emerged as a new paradigm for recommendation by converting interacted item history into language modeling. However, constrained by the limited context length of LLMs, existing approaches have to truncate item history in the prompt, focusing only on recent interactions and sacrificing the ability to model long-term history. To enable LLMs to model long histories, we pursue a concise embedding representation for items and sessions. In the LLM embedding space, we construct an item's embedding by aggregating its textual token embeddings; similarly, we construct a session's embedding by aggregating its item embeddings. While efficient, this way poses two challenges since it ignores the temporal significance of user interactions and LLMs do not natively interpret our custom embeddings. To overcome these, we propose PatchRec, a multi-grained patch training method consisting of two stages: (1) Patch Pre-training, which familiarizes LLMs with aggregated embeddings -- patches, and (2) Patch Fine-tuning, which enables LLMs to capture time-aware significance in interaction history. Extensive experiments show that PatchRec effectively models longer behavior histories with improved efficiency. This work facilitates the practical use of LLMs for modeling long behavior histories. Codes are available at https://github.com/ljy0ustc/PatchRec.

Multi-Grained Patch Training for Efficient LLM-based Recommendation

TL;DR

PatchRec introduces a multi-grained patch training framework for LLM-based recommendations to model long-term user histories within limited context windows. It hierarchically compresses item histories into item patches and session patches, and employs a two-stage training process—Patch Pre-training and Patch Fine-tuning—to teach the model to interpret compressed representations and to assign time-aware significance to interactions. The approach yields substantial gains in recommendation accuracy (e.g., HR@20 improvements up to 32%) while dramatically reducing input-token usage (compression ratios up to multi-fold), outperforming TALLRec and other baselines on MovieLens and Goodreads datasets. These results demonstrate that PatchRec enables practical, long-horizon LLM-based sequential recommendations with improved efficiency and robustness for real-world applications.

Abstract

Large Language Models (LLMs) have emerged as a new paradigm for recommendation by converting interacted item history into language modeling. However, constrained by the limited context length of LLMs, existing approaches have to truncate item history in the prompt, focusing only on recent interactions and sacrificing the ability to model long-term history. To enable LLMs to model long histories, we pursue a concise embedding representation for items and sessions. In the LLM embedding space, we construct an item's embedding by aggregating its textual token embeddings; similarly, we construct a session's embedding by aggregating its item embeddings. While efficient, this way poses two challenges since it ignores the temporal significance of user interactions and LLMs do not natively interpret our custom embeddings. To overcome these, we propose PatchRec, a multi-grained patch training method consisting of two stages: (1) Patch Pre-training, which familiarizes LLMs with aggregated embeddings -- patches, and (2) Patch Fine-tuning, which enables LLMs to capture time-aware significance in interaction history. Extensive experiments show that PatchRec effectively models longer behavior histories with improved efficiency. This work facilitates the practical use of LLMs for modeling long behavior histories. Codes are available at https://github.com/ljy0ustc/PatchRec.
Paper Structure (28 sections, 7 equations, 9 figures, 4 tables)

This paper contains 28 sections, 7 equations, 9 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Method
  4. Hierarchical Patching
  5. Item Patch Construction.
  6. Session Patch Construction.
  7. Challenges of Patch Training
  8. PatchRec
  9. Patch Pre-training
  10. Patch Fine-tuning
  11. Experiments
  12. Experimental Settings
  13. Main Performance Comparison
  14. Baselines
  15. Results
  16. ...and 13 more sections

Figures (9)

  • Figure 1: Most LLM4SR works consider the latest-$K$ interactions in the complete user interaction sequence as the truncated historical item sequence fed to LLMs.
  • Figure 2: Performance-efficiency trade-off for TALLRec, Pure-Item/Session, and PatchRec-I/S on MovieLens-1M. PatchRec variants demonstrate improvements in both metrics compared to the baseline. The shaded region highlights the area above the performance-efficiency trade-off curve, representing desirable configurations.
  • Figure 3: Performance of LLM-based recommender on Movielens-1M at different numbers of items in the truncated interaction sequence. The number of tokens in the historical item sequence ranges from 39 to 552, excluding the tokens of task description.
  • Figure 4: Hierarchical compression. The textual tokens of an item title are aggregated into a compact item patch. Then, several adjacent item patches are further compressed into a denser session patch.
  • Figure 5: Two-stage training framework of PatchRec. In the patch pre-training stage, we augment each uncompressed sequence $s$ with a compressed version $s'$, where each item is independently and randomly compressed from raw textual tokens into an item patch, in order to build connections between item patches and textual item titles. During the training process, the degree of compression gradually increases from 0 to 1. In the patch finetuning stage, we fine-tune the LLMs with time-aware compressed sequences, allowing the LLM to become familiar with a mixed space of various compression granularities in downstream usage. For each sequence, interactions that occur earlier are compressed to a greater extent, while interactions that occur later are compressed to a lesser extent. PFT-S can achieve higher compression ratio with denser session patches.
  • ...and 4 more figures