Learning Item Representations Directly from Multimodal Features for Effective Recommendation

TL;DR

This paper tackles the gradient bias in multimodal recommender systems that overemphasizes multimodal features at the expense of item ID embeddings. It introduces LIRDRec, a model that learns item representations directly from multimodal features through a multimodal feature transformation, graph learning, and a progressive weight copying mechanism to balance modality contributions. The approach, enhanced by MLLMs for image-to-text conversion and LLM-derived embeddings, achieves state-of-the-art results across five real-world datasets, with notable gains in NDCG@20, especially when multimodal features are enriched by large-language models. The work highlights the practical impact of leveraging rich multimodal information for faster, more robust recommendations, and points to future extensions in sequential settings and MLLM-based semantic augmentation.

Abstract

Conventional multimodal recommender systems predominantly leverage Bayesian Personalized Ranking (BPR) optimization to learn item representations by amalgamating item identity (ID) embeddings with multimodal features. Nevertheless, our empirical and theoretical findings unequivocally demonstrate a pronounced optimization gradient bias in favor of acquiring representations from multimodal features over item ID embeddings. As a consequence, item ID embeddings frequently exhibit suboptimal characteristics despite the convergence of multimodal feature parameters. Given the rich informational content inherent in multimodal features, in this paper, we propose a novel model (i.e., LIRDRec) that learns item representations directly from these features to augment recommendation performance. Recognizing that features derived from each modality may capture disparate yet correlated aspects of items, we propose a multimodal transformation mechanism, integrated with modality-specific encoders, to effectively fuse features from all modalities. Moreover, to differentiate the influence of diverse modality types, we devise a progressive weight copying fusion module within LIRDRec. This module incrementally learns the weight assigned to each modality in synthesizing the final user or item representations. Finally, we utilize the powerful visual understanding of Multimodal Large Language Models (MLLMs) to convert the item images into texts and extract semantics embeddings upon the texts via LLMs. Empirical evaluations conducted on five real-world datasets validate the superiority of our approach relative to competing baselines. It is worth noting the proposed model, equipped with embeddings extracted from MLLMs and LLMs, can further improve the recommendation accuracy of NDCG@20 by an average of 4.21% compared to the original embeddings.

Figures (7)

• Figure 1: Comparison of training losses between ID embedding and multimodal feature (MM) learning in VBPR VBPR2016AAAI and FREEDOM FREEDOM2023MM. The magnitude of the gradient flow in MM learning exhibits a steeper and more pronounced decay compared to the gradients observed in ID embedding optimization.
• Figure 2: An overview of LIRDRec. Instead of utilizing ID embeddings to represent items, LIRDRec directly derives item representations from multimodal features.
• Figure 3: The proposed LIRDRec can quickly boost startup recommendation performance for both datasets.
• Figure 4: Performance of LIRDRec compared with representative baselines under cold-start settings.
• Figure 5: Performance of LIRDRec with regard to different regularization coefficient $\lambda$ and decay rate $\tau$.

Theorems & Definitions (4)

• Theorem 1
• proof
• Lemma 2
• proof