Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Trie-Aware Transformers for Generative Recommendation

Zhenxiang Xu, Jiawei Chen, Sirui Chen, Yong He, Jieyu Yang, Chuan Yuan, Ke Ding, Can Wang

TL;DR

TrieRec is a trie-aware generative recommendation method that augments Transformers with structural inductive biases via two positional encodings, which are model-agnostic, efficient, and hyperparameter-free.

Abstract

Generative recommendation (GR) aligns with advances in generative AI by casting next-item prediction as token-level generation rather than score-based ranking. Most GR methods adopt a two-stage pipeline: (i) \textit{item tokenization}, which maps each item to a sequence of discrete, hierarchically organized tokens; and (ii) \textit{autoregressive generation}, which predicts the next item's tokens conditioned on the tokens of user's interaction history. Although hierarchical tokenization induces a prefix tree (trie) over items, standard autoregressive modeling with conventional Transformers often flattens item tokens into a linear stream and overlooks the underlying topology. To address this, we propose TrieRec, a trie-aware generative recommendation method that augments Transformers with structural inductive biases via two positional encodings. First, a \textit{trie-aware absolute positional encoding} aggregates a token's (node's) local structural context (\eg depth, ancestors, and descendants) into the token representation. Second, a \textit{topology-aware relative positional encoding} injects pairwise structural relations into self-attention to capture topology-induced semantic relatedness. TrieRec is also model-agnostic, efficient, and hyperparameter-free. In our experiments, we implement TrieRec within three representative GR backbones, achieving notably improvements of 8.83\% on average across four real-world datasets.

Trie-Aware Transformers for Generative Recommendation

TL;DR

TrieRec is a trie-aware generative recommendation method that augments Transformers with structural inductive biases via two positional encodings, which are model-agnostic, efficient, and hyperparameter-free.

Abstract

Generative recommendation (GR) aligns with advances in generative AI by casting next-item prediction as token-level generation rather than score-based ranking. Most GR methods adopt a two-stage pipeline: (i) \textit{item tokenization}, which maps each item to a sequence of discrete, hierarchically organized tokens; and (ii) \textit{autoregressive generation}, which predicts the next item's tokens conditioned on the tokens of user's interaction history. Although hierarchical tokenization induces a prefix tree (trie) over items, standard autoregressive modeling with conventional Transformers often flattens item tokens into a linear stream and overlooks the underlying topology. To address this, we propose TrieRec, a trie-aware generative recommendation method that augments Transformers with structural inductive biases via two positional encodings. First, a \textit{trie-aware absolute positional encoding} aggregates a token's (node's) local structural context (\eg depth, ancestors, and descendants) into the token representation. Second, a \textit{topology-aware relative positional encoding} injects pairwise structural relations into self-attention to capture topology-induced semantic relatedness. TrieRec is also model-agnostic, efficient, and hyperparameter-free. In our experiments, we implement TrieRec within three representative GR backbones, achieving notably improvements of 8.83\% on average across four real-world datasets.
Paper Structure (18 sections, 6 equations, 3 figures, 4 tables)

This paper contains 18 sections, 6 equations, 3 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Preliminary
  3. Methodology
  4. Trie‑aware Absolute Positional Encoding (TAE)
  5. Topology-aware Relative Positional Encoding (TRE)
  6. Discussions
  7. Experiments
  8. Experimental Setup
  9. Datasets
  10. Metrics
  11. Baselines
  12. Implementation Details
  13. Overall Performance (RQ1)
  14. Ablation Study(RQ2)
  15. Efficiency Analysis (RQ3)
  16. ...and 3 more sections

Figures (3)

  • Figure 1: Generative recommendation pipeline. Item tokenization maps items to semantic token sequences via hierarchical quantization, forming a trie structure. Autoregressive generation employs a Transformer to predict the next item's token sequence conditioned on user interaction history.
  • Figure 2: The illustration of our proposed trie-aware absolute positional encoding and topology-aware relative positional encoding in TrieRec.
  • Figure 3: Efficiency comparison between TrieRec and baseline LETTER on Beauty and Toy datasets. The left panel shows the total training time, and the right panel shows the inference latency.