Beyond Unimodal Boundaries: Generative Recommendation with Multimodal Semantics

TL;DR

The paper addresses the limitations of unimodal content in generative recommendation by formalizing Multimodal Generative Recommendation (MGR) and comparing early and late fusion strategies. It finds that naive multimodal fusion struggles with modality dominance and cross-modal ID mapping, and introduces MGR-LF++—an enhanced late-fusion framework featuring Contrastive Modality Alignment and modality-transition tokens. Empirical results on three Amazon domains show that MGR-LF++ yields up to approximately 20% improvements over unimodal baselines and naive multimodal approaches, with ablations highlighting the importance of semantic ID length and codebook size. The work advances understanding of how to leverage multimodal content in autoregressive recommendation and provides concrete mechanisms to align and integrate cross-modal semantic IDs for robust retrieval. These findings have practical implications for building more accurate and versatile recommender systems in multimodal environments.

Abstract

Generative recommendation (GR) has become a powerful paradigm in recommendation systems that implicitly links modality and semantics to item representation, in contrast to previous methods that relied on non-semantic item identifiers in autoregressive models. However, previous research has predominantly treated modalities in isolation, typically assuming item content is unimodal (usually text). We argue that this is a significant limitation given the rich, multimodal nature of real-world data and the potential sensitivity of GR models to modality choices and usage. Our work aims to explore the critical problem of Multimodal Generative Recommendation (MGR), highlighting the importance of modality choices in GR nframeworks. We reveal that GR models are particularly sensitive to different modalities and examine the challenges in achieving effective GR when multiple modalities are available. By evaluating design strategies for effectively leveraging multiple modalities, we identify key challenges and introduce MGR-LF++, an enhanced late fusion framework that employs contrastive modality alignment and special tokens to denote different modalities, achieving a performance improvement of over 20% compared to single-modality alternatives.

Figures (5)

• Figure 2: Naive strategies for extending unimodal generative recommendation to multimodal scenarios. Early fusion (MGR-EF) generates a unified list of semantic IDs capturing the semantics across all modalities. Late fusion (MGR-LF) generates separate semantic IDs for each modality and combines them after generation.
• Figure 3: Key challenges in the MGR problem. (a) Modality Sensitivity: Despite both items being transformers, they are distinct products. The semantic ID generated through early fusion fails to capture visual differences, resulting in information loss. (b) Modality Correspondence: The item's textual semantic ID is [0,2], with a corresponding visual semantic ID of [1,7]. While the sequential recommender correctly predicts the item as a transformer (by accurately predicting the textual semantic ID [0,2]), it fails to map this to the corresponding visual semantic ID [1,7].
• Figure 4: Multimodality is needed: Semantic IDs of each modality captures different information. Visualization of items with the same first semantic ID per modality wrt. the pivot example on Toys. (a) focuses on capturing items with similar textual semantics with various visual lookings while (b) focuses on capturing items that look similar to each other visually.
• Figure 5: Ablation Study on our two design choices across datasets. Both of our proposed designs: special token for cross-modal indication and contrastive modality are important for the MGR problem.
• Figure 6: Influence of (a) semantic ID length and (b) codebook size on MGR-LF++ on Sports. For larger datasets, it is important to have a longer semantic ID length. Setting codebook size to 256 gets the best performance.