Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SpecTran: Spectral-Aware Transformer-based Adapter for LLM-Enhanced Sequential Recommendation

Yu Cui, Feng Liu, Zhaoxiang Wang, Changwang Zhang, Jun Wang, Can Wang, Jiawei Chen

TL;DR

This work tackles the gap between high-dimensional LLM language embeddings and low-dimensional item embeddings in sequential recommendation by introducing SpecTran, a spectral-aware transformer-based adapter. By operating in the spectral domain and employing a learnable spectral-position encoding, SpecTran attends across the full spectrum to aggregate informative components while mitigating spectral collapse. Empirical results across four real-world datasets and three backbones show an average improvement of $9.17\%$ over strong baselines, with ablations confirming the importance of spectral attention and the Taylor-expansion-based weighting of principal components. The approach is lightweight, model-agnostic, and readily pluggable into existing SR backbones, offering practical gains in accuracy with modest computational overhead.

Abstract

Traditional sequential recommendation (SR) models learn low-dimensional item ID embeddings from user-item interactions, often overlooking textual information such as item titles or descriptions. Recent advances in Large Language Models (LLMs) have inspired a surge of research that encodes item textual information with high-dimensional semantic embeddings, and designs transformation methods to inject such embeddings into SR models. These embedding transformation strategies can be categorized into two types, both of which exhibits notable drawbacks: 1) adapter-based methods suffer from pronounced dimension collapse, concentrating information into a few dominant dimensions; 2) SVD-based methods are rigid and manual, considering only a few principal spectral components while discarding rich information in the remaining spectrum. To address these limitations, we propose SpecTran, a spectral-aware transformer-based adapter that operates in the spectral domain, attending to the full spectrum to select and aggregates informative components. A learnable spectral-position encoding injects singular-value cues as an inductive bias, guiding attention toward salient spectral components and promoting diversity across embedding dimensions. Across four real-world datasets and three SR backbones, it consistently outperforms strong baselines, achieving an average improvement of 9.17%.

SpecTran: Spectral-Aware Transformer-based Adapter for LLM-Enhanced Sequential Recommendation

TL;DR

This work tackles the gap between high-dimensional LLM language embeddings and low-dimensional item embeddings in sequential recommendation by introducing SpecTran, a spectral-aware transformer-based adapter. By operating in the spectral domain and employing a learnable spectral-position encoding, SpecTran attends across the full spectrum to aggregate informative components while mitigating spectral collapse. Empirical results across four real-world datasets and three backbones show an average improvement of over strong baselines, with ablations confirming the importance of spectral attention and the Taylor-expansion-based weighting of principal components. The approach is lightweight, model-agnostic, and readily pluggable into existing SR backbones, offering practical gains in accuracy with modest computational overhead.

Abstract

Traditional sequential recommendation (SR) models learn low-dimensional item ID embeddings from user-item interactions, often overlooking textual information such as item titles or descriptions. Recent advances in Large Language Models (LLMs) have inspired a surge of research that encodes item textual information with high-dimensional semantic embeddings, and designs transformation methods to inject such embeddings into SR models. These embedding transformation strategies can be categorized into two types, both of which exhibits notable drawbacks: 1) adapter-based methods suffer from pronounced dimension collapse, concentrating information into a few dominant dimensions; 2) SVD-based methods are rigid and manual, considering only a few principal spectral components while discarding rich information in the remaining spectrum. To address these limitations, we propose SpecTran, a spectral-aware transformer-based adapter that operates in the spectral domain, attending to the full spectrum to select and aggregates informative components. A learnable spectral-position encoding injects singular-value cues as an inductive bias, guiding attention toward salient spectral components and promoting diversity across embedding dimensions. Across four real-world datasets and three SR backbones, it consistently outperforms strong baselines, achieving an average improvement of 9.17%.
Paper Structure (28 sections, 11 equations, 2 figures, 6 tables)

This paper contains 28 sections, 11 equations, 2 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Task Formulation
  4. Language Embedding Enhancement
  5. Semantic Encoding
  6. Embedding Transformation
  7. Embedding Fusion
  8. Methodology
  9. Spectral-Aware Transformer-based Adapter
  10. Spectral-Aware Positional Encoding
  11. Experiments
  12. Experimental Settings
  13. Datasets
  14. Backbones
  15. Baselines
  16. ...and 13 more sections

Figures (2)

  • Figure 1: The distribution of cumulative singular values on the Toy and Beauty dataset. Similar results are observed on other datasets.
  • Figure 2: The overall framework of the proposed SpecTran.