Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multimodal Pretraining, Adaptation, and Generation for Recommendation: A Survey

Qijiong Liu, Jieming Zhu, Yanting Yang, Quanyu Dai, Zhaocheng Du, Xiao-Ming Wu, Zhou Zhao, Rui Zhang, Zhenhua Dong

TL;DR

Traditional recommender systems rely on IDs and categorical features, often missing rich multimodal item content. This survey organizes the landscape around multimodal pretraining, adaptation, and generation, detailing self-supervised and content-aware pretraining, four adaptation strategies, and text/image/video/personalized generation for recommendations. It covers applications across e-commerce, advertising, news, video, and music, and highlights open challenges such as hierarchical fusion, cross-domain alignment, foundation models, AIGC integration, and efficiency. The work provides a resource-oriented guide to building content-aware recommender systems using multimodal models and generation techniques.

Abstract

Personalized recommendation serves as a ubiquitous channel for users to discover information tailored to their interests. However, traditional recommendation models primarily rely on unique IDs and categorical features for user-item matching, potentially overlooking the nuanced essence of raw item contents across multiple modalities such as text, image, audio, and video. This underutilization of multimodal data poses a limitation to recommender systems, especially in multimedia services like news, music, and short-video platforms. The recent advancements in large multimodal models offer new opportunities and challenges in developing content-aware recommender systems. This survey seeks to provide a comprehensive exploration of the latest advancements and future trajectories in multimodal pretraining, adaptation, and generation techniques, as well as their applications in enhancing recommender systems. Furthermore, we discuss current open challenges and opportunities for future research in this dynamic domain. We believe that this survey, alongside the curated resources, will provide valuable insights to inspire further advancements in this evolving landscape.

Multimodal Pretraining, Adaptation, and Generation for Recommendation: A Survey

TL;DR

Traditional recommender systems rely on IDs and categorical features, often missing rich multimodal item content. This survey organizes the landscape around multimodal pretraining, adaptation, and generation, detailing self-supervised and content-aware pretraining, four adaptation strategies, and text/image/video/personalized generation for recommendations. It covers applications across e-commerce, advertising, news, video, and music, and highlights open challenges such as hierarchical fusion, cross-domain alignment, foundation models, AIGC integration, and efficiency. The work provides a resource-oriented guide to building content-aware recommender systems using multimodal models and generation techniques.

Abstract

Personalized recommendation serves as a ubiquitous channel for users to discover information tailored to their interests. However, traditional recommendation models primarily rely on unique IDs and categorical features for user-item matching, potentially overlooking the nuanced essence of raw item contents across multiple modalities such as text, image, audio, and video. This underutilization of multimodal data poses a limitation to recommender systems, especially in multimedia services like news, music, and short-video platforms. The recent advancements in large multimodal models offer new opportunities and challenges in developing content-aware recommender systems. This survey seeks to provide a comprehensive exploration of the latest advancements and future trajectories in multimodal pretraining, adaptation, and generation techniques, as well as their applications in enhancing recommender systems. Furthermore, we discuss current open challenges and opportunities for future research in this dynamic domain. We believe that this survey, alongside the curated resources, will provide valuable insights to inspire further advancements in this evolving landscape.
Paper Structure (16 sections, 1 figure)

This paper contains 16 sections, 1 figure.

Table of Contents

  1. Introduction
  2. Multimodal Pretraining for Recommendation
  3. Self-supervised Pretraining Paradigms
  4. Content-aware Pretraining for Recommendation
  5. Multimodal Adaption for Recommendation
  6. Representation Transfer
  7. Model Finetuning
  8. Adapter Tuning
  9. Prompt Tuning
  10. Multimodal Generation for Recommendation
  11. Text Generation
  12. Image and Video Generation
  13. Personalized Generation
  14. Applications
  15. Challenges and Opportunities
  16. ...and 1 more sections

Figures (1)

  • Figure 1: An overview of multimodal pretraining, adaptation, and generation tasks for recommendation.