FusID: Modality-Fused Semantic IDs for Generative Music Recommendation
Haven Kim, Yupeng Hou, Julian McAuley
TL;DR
FusID addresses redundancy and missing inter-modal interactions in semantic-ID based generative music recommendations by fusing multiple modalities into a unified embedding $E \in \mathbb{R}^{n \times d}$, which is discretized with product quantization using $K=1024$ clusters per subspace to form token sequences $(c_{1,x_1}, \dots, c_{n,x_n})$. The model optimizes a contrastive loss to align frequently co-occurring items and a block-wise cross-covariance regularization inspired by VICReg to maintain diversity among the $n$ sub-embeddings. On MPD, FusID achieves zero ID conflicts, full codebook utilization, and superior MRR and Recall@$k$ compared to baselines SASRec and TalkPlay, demonstrating the practicality of zero-conflict, multimodal semantic IDs for scalable generative recommendation. These results highlight the value of joint multimodal encoding and structured quantization for robust, efficient generative recommender systems in music.
Abstract
Generative recommendation systems have achieved significant advances by leveraging semantic IDs to represent items. However, existing approaches that tokenize each modality independently face two critical limitations: (1) redundancy across modalities that reduces efficiency, and (2) failure to capture inter-modal interactions that limits item representation. We introduce FusID, a modality-fused semantic ID framework that addresses these limitations through three key components: (i) multimodal fusion that learns unified representations by jointly encoding information across modalities, (ii) representation learning that brings frequently co-occurring item embeddings closer while maintaining distinctiveness and preventing feature redundancy, and (iii) product quantization that converts the fused continuous embeddings into multiple discrete tokens to mitigate ID conflict. Evaluated on a multimodal next-song recommendation (i.e., playlist continuation) benchmark, FusID achieves zero ID conflicts, ensuring that each token sequence maps to exactly one song, mitigates codebook underutilization, and outperforms baselines in terms of MRR and Recall@k (k = 1, 5, 10, 20).