Multi-Aspect Cross-modal Quantization for Generative Recommendation

TL;DR

This work tackles Generative Recommendation by integrating multimodal information into semantic ID learning and GR training. It introduces MACRec, a two-part framework that performs cross-modal item quantization with multimodal contrastive learning to generate hierarchically meaningful semantic IDs and employs both implicit latent-space and explicit cross-modal alignments during Seq2Seq generation. Key contributions include dual-modality pseudo-labels, layer-wise cross-modal residual quantization with reconstruction alignment, and multi-aspect alignment strategies that combine latent-space and task-driven signals. Empirical results on three Amazon-based datasets show MACRec consistently outperforms state-of-the-art multimodal GR methods, reduces item collisions, and achieves more balanced codebook utilization, highlighting the practical impact for scalable, accurate multimodal recommendations.

Abstract

Generative Recommendation (GR) has emerged as a new paradigm in recommender systems. This approach relies on quantized representations to discretize item features, modeling users' historical interactions as sequences of discrete tokens. Based on these tokenized sequences, GR predicts the next item by employing next-token prediction methods. The challenges of GR lie in constructing high-quality semantic identifiers (IDs) that are hierarchically organized, minimally conflicting, and conducive to effective generative model training. However, current approaches remain limited in their ability to harness multimodal information and to capture the deep and intricate interactions among diverse modalities, both of which are essential for learning high-quality semantic IDs and for effectively training GR models. To address this, we propose Multi-Aspect Cross-modal quantization for generative Recommendation (MACRec), which introduces multimodal information and incorporates it into both semantic ID learning and generative model training from different aspects. Specifically, we first introduce cross-modal quantization during the ID learning process, which effectively reduces conflict rates and thus improves codebook usability through the complementary integration of multimodal information. In addition, to further enhance the generative ability of our GR model, we incorporate multi-aspect cross-modal alignments, including the implicit and explicit alignments. Finally, we conduct extensive experiments on three well-known recommendation datasets to demonstrate the effectiveness of our proposed method.

Figures (4)

• Figure 1: Visualization examples of different modality embeddings on Instrument.
• Figure 2: Overall architecture of MACRec. Left: Cross-modal Item Quantization, including Dual-modality Pseudo-label Generation, Cross-modal Quantization, and Cross-modal Reconstruction Alignment, aiming to generate high-quality semantic IDs across different modalities. Right: Generative Recommendation with Multi-aspect Alignment, aligning features from different modalities via implicit alignment, explicit alignment, and training the GR model with Seq2Seq task.
• Figure 3: Performance of MACRec over different hyper-parameters on Instruments.
• Figure 4: Code assignment distribution on the 2-th RQ layer.