Tokenize Once, Recommend Anywhere: Unified Item Tokenization for Multi-domain LLM-based Recommendation
Yu Hou, Won-Yong Shin
TL;DR
UniTok addresses the challenge of unified, cross-domain item tokenization for LLM-based recommendations by integrating a customized mixture-of-experts (TokenMoE) with per-domain and shared codebooks, plus a mutual information calibration to balance semantics across domains. The architecture yields a shared latent space from which discrete tokens are derived via residual quantization, enabling effective cross-domain generalization without per-domain retraining. The authors prove theoretical benefits (higher token-space entropy, lower quantization error, and bounded cross-domain performance variability) and demonstrate substantial empirical gains (up to 51.89% NDCG@10 improvements) and significant parameter efficiency (≈9.63× fewer trainable parameters) across ten real-world domains, with strong zero-shot performance on unseen domains. Overall, UniTok offers a scalable, generalizable tokenization framework that enhances LLM-based recommendations by bridging item semantics and language representations while reducing deployment overhead.
Abstract
Large language model (LLM)-based recommender systems have achieved high-quality performance by bridging the discrepancy between the item space and the language space through item tokenization. However, existing item tokenization methods typically require training separate models for each item domain, limiting generalization. Moreover, the diverse distributions and semantics across item domains make it difficult to construct a unified tokenization that preserves domain-specific information. To address these challenges, we propose UniTok, a Unified item Tokenization framework that integrates our own mixture-of-experts (MoE) architecture with a series of codebooks to convert items into discrete tokens, enabling scalable tokenization while preserving semantic information across multiple item domains. Specifically, items from different domains are first projected into a unified latent space through a shared encoder. They are then routed to domain-specific experts to capture the unique semantics, while a shared expert, which is always active, encodes common knowledge transferable across domains. Additionally, to mitigate semantic imbalance across domains, we present a mutual information calibration mechanism, which guides the model towards retaining similar levels of semantic information for each domain. Comprehensive experiments on wide-ranging real-world datasets demonstrate that the proposed UniTok framework is (a) highly effective: achieving up to 51.89% improvements over strong benchmarks, (b) theoretically sound: showing the analytical validity of our architectural design and optimization; and (c) highly generalizable: demonstrating robust performance across diverse domains without requiring per-domain retraining, a capability not supported by existing baselines.