Pre-training Generative Recommender with Multi-Identifier Item Tokenization

TL;DR

MTGRec tackles the limitations of one-to-one item tokenization in generative recommenders by introducing multi-identifier item tokenization via semantically related RQ-VAE checkpoints and a curriculum pre-training scheme guided by data influence. The approach augments training data with multiple token sequences per item and dynamically adapts data sampling to emphasize informative tokenizers, followed by fine-tuning on a single tokenizer for precise item identification. Empirical results on three Amazon subsets show MTGRec consistently outperforms traditional and generative baselines, with notable improvements on long-tail items and scalable performance as model size grows. This work advances scalable, semantically rich generative recommendation by combining tokenization diversity with data-aware pre-training.

Abstract

Generative recommendation autoregressively generates item identifiers to recommend potential items. Existing methods typically adopt a one-to-one mapping strategy, where each item is represented by a single identifier. However, this scheme poses issues, such as suboptimal semantic modeling for low-frequency items and limited diversity in token sequence data. To overcome these limitations, we propose MTGRec, which leverages Multi-identifier item Tokenization to augment token sequence data for Generative Recommender pre-training. Our approach involves two key innovations: multi-identifier item tokenization and curriculum recommender pre-training. For multi-identifier item tokenization, we leverage the RQ-VAE as the tokenizer backbone and treat model checkpoints from adjacent training epochs as semantically relevant tokenizers. This allows each item to be associated with multiple identifiers, enabling a single user interaction sequence to be converted into several token sequences as different data groups. For curriculum recommender pre-training, we introduce a curriculum learning scheme guided by data influence estimation, dynamically adjusting the sampling probability of each data group during recommender pre-training. After pre-training, we fine-tune the model using a single tokenizer to ensure accurate item identification for recommendation. Extensive experiments on three public benchmark datasets demonstrate that MTGRec significantly outperforms both traditional and generative recommendation baselines in terms of effectiveness and scalability.

Figures (5)

• Figure 1: The overall framework of MTGRec with two key techniques. (i) We utilize RQ-VAE checkpoints of adjacent epochs as semantically relevant item tokenizers and tokenize an item sequence into multiple token sequences. (ii) We propose a data curriculum scheme based on data influence estimation, which is implemented through first-order gradient approximation.
• Figure 2: Performance Comparison w.r.t. Model Scale. The x-axis coordinates are the number of encoder and decoder layers in the generative recommender. All reported results for MTGRec are the best results obtained using various numbers of tokenizers.
• Figure 3: Performance Comparison w.r.t. Tokenizer Number on the Instrument and Scientific datasets.
• Figure 4: Performance Comparison w.r.t. Temperature Coefficient on the Instrument and Scientific datasets.
• Figure 5: Performance Comparison w.r.t. Long-tail Items on the Instrument and Scientific datasets.. The bar graph illustrates the number of interactions in the test data for each group, while the line chart displays the improvement ratios for Recall@10 in comparison to TIGER.