Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

RecGOAT: Graph Optimal Adaptive Transport for LLM-Enhanced Multimodal Recommendation with Dual Semantic Alignment

Yuecheng Li, Hengwei Ju, Zeyu Song, Wei Yang, Chi Lu, Peng Jiang, Kun Gai

TL;DR

RecGOAT tackles the semantic heterogeneity between world-knowledge encoded LM representations and sparse ID signals in multimodal recommendation. It introduces a dual semantic alignment framework combining intra-modal graph learning with LM-enhanced modalities and cross-modal instance-level CMCL plus distribution-level OT via OAT, with a learnable residual transport and fusion to a unified item representation. Theoretical analysis provides bounds guaranteeing alignment consistency and fusion comprehensiveness; Empirical results on three public datasets and a large online platform show state-of-the-art performance and scalability. This approach offers a principled, scalable path to integrate LLMs into production-grade multimodal recommendations.

Abstract

Multimodal recommendation systems typically integrates user behavior with multimodal data from items, thereby capturing more accurate user preferences. Concurrently, with the rise of large models (LMs), multimodal recommendation is increasingly leveraging their strengths in semantic understanding and contextual reasoning. However, LM representations are inherently optimized for general semantic tasks, while recommendation models rely heavily on sparse user/item unique identity (ID) features. Existing works overlook the fundamental representational divergence between large models and recommendation systems, resulting in incompatible multimodal representations and suboptimal recommendation performance. To bridge this gap, we propose RecGOAT, a novel yet simple dual semantic alignment framework for LLM-enhanced multimodal recommendation, which offers theoretically guaranteed alignment capability. RecGOAT first employs graph attention networks to enrich collaborative semantics by modeling item-item, user-item, and user-user relationships, leveraging user/item LM representations and interaction history. Furthermore, we design a dual-granularity progressive multimodality-ID alignment framework, which achieves instance-level and distribution-level semantic alignment via cross-modal contrastive learning (CMCL) and optimal adaptive transport (OAT), respectively. Theoretically, we demonstrate that the unified representations derived from our alignment framework exhibit superior semantic consistency and comprehensiveness. Extensive experiments on three public benchmarks show that our RecGOAT achieves state-of-the-art performance, empirically validating our theoretical insights. Additionally, the deployment on a large-scale online advertising platform confirms the model's effectiveness and scalability in industrial recommendation scenarios. Code available at https://github.com/6lyc/RecGOAT-LLM4Rec.

RecGOAT: Graph Optimal Adaptive Transport for LLM-Enhanced Multimodal Recommendation with Dual Semantic Alignment

TL;DR

RecGOAT tackles the semantic heterogeneity between world-knowledge encoded LM representations and sparse ID signals in multimodal recommendation. It introduces a dual semantic alignment framework combining intra-modal graph learning with LM-enhanced modalities and cross-modal instance-level CMCL plus distribution-level OT via OAT, with a learnable residual transport and fusion to a unified item representation. Theoretical analysis provides bounds guaranteeing alignment consistency and fusion comprehensiveness; Empirical results on three public datasets and a large online platform show state-of-the-art performance and scalability. This approach offers a principled, scalable path to integrate LLMs into production-grade multimodal recommendations.

Abstract

Multimodal recommendation systems typically integrates user behavior with multimodal data from items, thereby capturing more accurate user preferences. Concurrently, with the rise of large models (LMs), multimodal recommendation is increasingly leveraging their strengths in semantic understanding and contextual reasoning. However, LM representations are inherently optimized for general semantic tasks, while recommendation models rely heavily on sparse user/item unique identity (ID) features. Existing works overlook the fundamental representational divergence between large models and recommendation systems, resulting in incompatible multimodal representations and suboptimal recommendation performance. To bridge this gap, we propose RecGOAT, a novel yet simple dual semantic alignment framework for LLM-enhanced multimodal recommendation, which offers theoretically guaranteed alignment capability. RecGOAT first employs graph attention networks to enrich collaborative semantics by modeling item-item, user-item, and user-user relationships, leveraging user/item LM representations and interaction history. Furthermore, we design a dual-granularity progressive multimodality-ID alignment framework, which achieves instance-level and distribution-level semantic alignment via cross-modal contrastive learning (CMCL) and optimal adaptive transport (OAT), respectively. Theoretically, we demonstrate that the unified representations derived from our alignment framework exhibit superior semantic consistency and comprehensiveness. Extensive experiments on three public benchmarks show that our RecGOAT achieves state-of-the-art performance, empirically validating our theoretical insights. Additionally, the deployment on a large-scale online advertising platform confirms the model's effectiveness and scalability in industrial recommendation scenarios. Code available at https://github.com/6lyc/RecGOAT-LLM4Rec.
Paper Structure (27 sections, 3 theorems, 20 equations, 3 figures, 3 tables)

This paper contains 27 sections, 3 theorems, 20 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Multimodal Recommendation
  4. LM-enhanced Recommendation
  5. OT-based Distribution Alignment
  6. Methodology
  7. Intra‑modal: LM‑enhanced Modality Augmentation and Graph Learning
  8. Item-Item Multimodal Graph Representation Learning
  9. ID Embedding Learning from User-Item Interaction Graph
  10. User-User Graph Learning with LLM Contextual Enhancement
  11. Cross‑modal: Dual-Granularity Alignment of LLM-enhanced Modalities and ID Signals
  12. Instance-level Alignment via Cross-Modal Contrastive Learning (CMCL)
  13. Distribution-level Alignment via Optimal Adaptive Transport (OAT)
  14. Theoretical Guarantees for Alignment Consistency and Fusion Comprehensiveness
  15. Problem Setup and Assumptions
  16. ...and 12 more sections

Key Result

Lemma 3.3

Let $\bm{z_i^m}$ and $\bm{z_i}$ be the $L_2$-normalized representations for modality $m$ and the unified representation for item $i$ ($i.e., \|\bm{z_i^m}\| = 1, \|\bm{z_i}\| = 1$), respectively. The expected pairwise distance is bounded by the contrastive loss: where $\mathcal{L}_{\text{CMCL}}$ is the InfoNCE-based cross-modal contrastive loss, $\tau$ is the temperature parameter, and $B$ is the

Figures (3)

  • Figure 1: Performance comparison between LM representations with or without alignment for recommendation systems on Baby Dataset. (A) Due to semantic heterogeneity, LM Representation w/o alignment leads to degradation in recommendation performance. (B) Through our dual‑granularity alignment, the semantic conflict is resolved, yielding performance improvements of 59% and 70%, respectively.
  • Figure 2: The overall framework of our RecGOAT.
  • Figure 3: Alignment Consistency and Fusion Comprehensiveness of RecGOAT on the Baby Dataset

Theorems & Definitions (3)

  • Lemma 3.3: Instance-level Distance Bound
  • Lemma 3.4: Modality-to-Unified Error Bound
  • Theorem 3.5: Alignment Consistency and Fusion Comprehensiveness of RecGOAT