Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Fine-grained Semantics Integration for Large Language Model-based Recommendation

Jiawen Feng, Xiaoyu Kong, Leheng Sheng, Bin Wu, Chao Yi, Feifang Yang, Xiang-Rong Sheng, Han Zhu, Xiang Wang, Jiancan Wu, Xiangnan He

TL;DR

TS-Rec comprises two key components: Semantic-Aware embedding Initialization, which initializes SID token embeddings by applying mean pooling to the pretrained embeddings of keywords extracted by a teacher model; and Token-level Semantic Alignment, which aligns individual tokens within the SID sequence with the shared semantics of the corresponding item clusters.

Abstract

Recent advances in Large Language Models (LLMs) have shifted in recommendation systems from the discriminative paradigm to the LLM-based generative paradigm, where the recommender autoregressively generates sequences of semantic identifiers (SIDs) for target items conditioned on historical interaction. While prevalent LLM-based recommenders have demonstrated performance gains by aligning pretrained LLMs between the language space and the SID space, modeling the SID space still faces two fundamental challenges: (1) Semantically Meaningless Initialization: SID tokens are randomly initialized, severing the semantic linkage between the SID space and the pretrained language space at start point, and (2) Coarse-grained Alignment: existing SFT-based alignment tasks primarily focus on item-level optimization, while overlooking the semantics of individual tokens within SID sequences.To address these challenges, we propose TS-Rec, which can integrate Token-level Semantics into LLM-based Recommenders. Specifically, TS-Rec comprises two key components: (1) Semantic-Aware embedding Initialization (SA-Init), which initializes SID token embeddings by applying mean pooling to the pretrained embeddings of keywords extracted by a teacher model; and (2) Token-level Semantic Alignment (TS-Align), which aligns individual tokens within the SID sequence with the shared semantics of the corresponding item clusters. Extensive experiments on two real-world benchmarks demonstrate that TS-Rec consistently outperforms traditional and generative baselines across all standard metrics. The results demonstrate that integrating fine-grained semantic information significantly enhances the performance of LLM-based generative recommenders.

Fine-grained Semantics Integration for Large Language Model-based Recommendation

TL;DR

TS-Rec comprises two key components: Semantic-Aware embedding Initialization, which initializes SID token embeddings by applying mean pooling to the pretrained embeddings of keywords extracted by a teacher model; and Token-level Semantic Alignment, which aligns individual tokens within the SID sequence with the shared semantics of the corresponding item clusters.

Abstract

Recent advances in Large Language Models (LLMs) have shifted in recommendation systems from the discriminative paradigm to the LLM-based generative paradigm, where the recommender autoregressively generates sequences of semantic identifiers (SIDs) for target items conditioned on historical interaction. While prevalent LLM-based recommenders have demonstrated performance gains by aligning pretrained LLMs between the language space and the SID space, modeling the SID space still faces two fundamental challenges: (1) Semantically Meaningless Initialization: SID tokens are randomly initialized, severing the semantic linkage between the SID space and the pretrained language space at start point, and (2) Coarse-grained Alignment: existing SFT-based alignment tasks primarily focus on item-level optimization, while overlooking the semantics of individual tokens within SID sequences.To address these challenges, we propose TS-Rec, which can integrate Token-level Semantics into LLM-based Recommenders. Specifically, TS-Rec comprises two key components: (1) Semantic-Aware embedding Initialization (SA-Init), which initializes SID token embeddings by applying mean pooling to the pretrained embeddings of keywords extracted by a teacher model; and (2) Token-level Semantic Alignment (TS-Align), which aligns individual tokens within the SID sequence with the shared semantics of the corresponding item clusters. Extensive experiments on two real-world benchmarks demonstrate that TS-Rec consistently outperforms traditional and generative baselines across all standard metrics. The results demonstrate that integrating fine-grained semantic information significantly enhances the performance of LLM-based generative recommenders.
Paper Structure (33 sections, 5 equations, 5 figures, 4 tables)

This paper contains 33 sections, 5 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. SID-based Generative Recommendation
  4. LLM-based Generative Recommendation
  5. Preliminaries
  6. Task Formulation
  7. Tokenization and Embedding Initialization
  8. Item Tokenization.
  9. Embedding Initialization.
  10. Supervised Fine Tuning for LLM-based Recommenders
  11. Method
  12. SA-Init: Semantic-Aware embedding Initialization
  13. TS-Align: Token-level Semantic Alignment
  14. Experiments
  15. Experimental Setting
  16. ...and 18 more sections

Figures (5)

  • Figure 1: Representations of SIDs before (i.e., left column) and after (i.e., right column) alignment. The randomly initialized SIDs are initially far away from the semantic space of the LLM (i.e., grey points), while SA-Init enables SIDs within the semantic space of the LLM. This makes the SID alignment of the randomly initialized one more difficult than SA-Init.
  • Figure 2: TS-Rec enables recommenders to have high-level and fine-grained understanding of token-level SIDs.
  • Figure 3: The overall framework of our TS-Rec. The pipeline consists of two complementary modules: (1) SA-Init, which initializes SID embeddings with semantic descriptions generated by an Semantic Extractor from clustered items; (2) TS-Align, which aligns each SID token with its semantics through multi-task instruction tuning.
  • Figure 4: Training dynamics of TS-Rec with different SA-Init depths on the Industrial and Office datasets. Configurations include Random Init, SA-Init 1 (first layer), SA-Init 2 (first two layers), and SA-Init 3 (all three layers).
  • Figure 5: Accuracy of generative retrieval tasks on the Industrial and Office datasets. ACC@1 measures Title2SID accuracy, and ACC@2 measures SID2Title accuracy.