Multimodal Pretraining and Generation for Recommendation: A Tutorial

TL;DR

The paper addresses the limitation of relying on unique IDs in recommender systems by advocating for the use of multimodal item content across text, image, audio, and video. It surveys multimodal pretraining and generation techniques and their applications to recommendation, organizing the discussion into four areas: sequence/text/audio multimodal pretraining, adaptation strategies, generative approaches for recommendations, and real-world industrial deployments with open challenges. The contribution lies in synthesizing recent advances into a cohesive tutorial that connects multimodal foundation models, self-supervised learning paradigms, and practical deployment considerations for recommender systems. This work provides a roadmap for researchers and practitioners to leverage multimodal signals and LLM-driven generation to improve personalization, cross-domain transfer, and scalable deployment in multimedia services.

Abstract

Personalized recommendation stands as a ubiquitous channel for users to explore information or items aligned with their interests. Nevertheless, prevailing recommendation models predominantly rely on unique IDs and categorical features for user-item matching. While this ID-centric approach has witnessed considerable success, it falls short in comprehensively grasping the essence of raw item contents across diverse modalities, such as text, image, audio, and video. This underutilization of multimodal data poses a limitation to recommender systems, particularly in the realm of multimedia services like news, music, and short-video platforms. The recent surge in pretraining and generation techniques presents both opportunities and challenges in the development of multimodal recommender systems. This tutorial seeks to provide a thorough exploration of the latest advancements and future trajectories in multimodal pretraining and generation techniques within the realm of recommender systems. The tutorial comprises three parts: multimodal pretraining, multimodal generation, and industrial applications and open challenges in the field of recommendation. Our target audience encompasses scholars, practitioners, and other parties interested in this domain. By providing a succinct overview of the field, we aspire to facilitate a swift understanding of multimodal recommendation and foster meaningful discussions on the future development of this evolving landscape.