Multimodal Generative Recommendation for Fusing Semantic and Collaborative Signals

TL;DR

The paper tackles the scalability of recommender systems on large item spaces by moving from item embeddings to discrete semantic codes and proposes MSCGRec, a multimodal generative framework that fuses semantic and collaborative signals. It introduces a self-supervised image quantization pipeline by integrating Residual Quantization with DINO, and treats collaborative features from sequential models as a separate modality, enabling richer representations. A constrained sequence learning mechanism restricts predictions to permissible code sequences, improving generalization and scalability, especially when handling missing modalities. Empirical results on three large real-world datasets show MSCGRec outperforms both traditional sequential and prior generative baselines, demonstrating the practical viability and effectiveness of multimodal generative recommendations at scale.

Abstract

Sequential recommender systems rank relevant items by modeling a user's interaction history and computing the inner product between the resulting user representation and stored item embeddings. To avoid the significant memory overhead of storing large item sets, the generative recommendation paradigm instead models each item as a series of discrete semantic codes. Here, the next item is predicted by an autoregressive model that generates the code sequence corresponding to the predicted item. However, despite promising ranking capabilities on small datasets, these methods have yet to surpass traditional sequential recommenders on large item sets, limiting their adoption in the very scenarios they were designed to address. To resolve this, we propose MSCGRec, a Multimodal Semantic and Collaborative Generative Recommender. MSCGRec incorporates multiple semantic modalities and introduces a novel self-supervised quantization learning approach for images based on the DINO framework. Additionally, MSCGRec fuses collaborative and semantic signals by extracting collaborative features from sequential recommenders and treating them as a separate modality. Finally, we propose constrained sequence learning that restricts the large output space during training to the set of permissible tokens. We empirically demonstrate on three large real-world datasets that MSCGRec outperforms both sequential and generative recommendation baselines and provide an extensive ablation study to validate the impact of each component.

Figures (1)

• Figure 1: Schematic overview of MSCGRec. (a) Each item in the history is represented by a joint encoding that encompasses all modalities. (b) Images are encoded by self-supervised quantization learning where the student embedding is encoded via residual quantization. (c) Sequence learning is performed by optimizing over permissible codes, where green nodes indicate the codes corresponding to the correct next item.