Multi-modal Generative Models in Recommendation System

TL;DR

This work surveys multimodal generative models for recommendations, addressing the problem of leveraging rich product and user data across modalities to overcome issues like cold-start and ambiguous queries. It juxtaposes contrastive approaches (e.g., CLIP) that align across modalities with generative models (GANs, VAEs, diffusion, and LLM-based systems) that explicitly model data distributions and support open-ended generation. Key contributions include a structured taxonomy of multimodal recommender models, insights into data alignment, and practical architectures for integration with collaborative filtering and content-based retrieval. The findings suggest that combining aligned multimodal representations with generative capabilities enables more flexible, interactive, and visually grounded recommendations, with significant potential for applications in e-commerce, visualization, marketing, streaming, and travel.

Abstract

Many recommendation systems limit user inputs to text strings or behavior signals such as clicks and purchases, and system outputs to a list of products sorted by relevance. With the advent of generative AI, users have come to expect richer levels of interactions. In visual search, for example, a user may provide a picture of their desired product along with a natural language modification of the content of the picture (e.g., a dress like the one shown in the picture but in red color). Moreover, users may want to better understand the recommendations they receive by visualizing how the product fits their use case, e.g., with a representation of how a garment might look on them, or how a furniture item might look in their room. Such advanced levels of interaction require recommendation systems that are able to discover both shared and complementary information about the product across modalities, and visualize the product in a realistic and informative way. However, existing systems often treat multiple modalities independently: text search is usually done by comparing the user query to product titles and descriptions, while visual search is typically done by comparing an image provided by the customer to product images. We argue that future recommendation systems will benefit from a multi-modal understanding of the products that leverages the rich information retailers have about both customers and products to come up with the best recommendations. In this chapter we review recommendation systems that use multiple data modalities simultaneously.

Figures (5)

• Figure 1: Contrastive pre-training used to train models such as CLIP. For each minibatch, the positive (diagonal) and negative (off-diagonal) pairs are used to compute the loss.
• Figure 2: Generative Adversarial Networks (GANs) are composed of a generator $G$ that generates a data point (e.g. an image) from a latent variable $Z$, and a discriminator $D$ which tries to determine if a data point is fake (synthesized by the generator) or real. Left: Standard GAN architecture for unconditional generation. Center: Conditional GAN architecture. Right: Multimodal GAN architecture.
• Figure 3: A Variational AutoEncoder (VAE) is composed of a probabilistic encoder that maps an input data point to a latent distribution from which a latent vector is sampled, and decoded with a probabilistic decoder with the objective of reconstructing the original input. Top: Standard VAE architecture for unconditional generation. Bottom left: Multimodal VAE architecture with a shared encoder and two unimodal decoders. Bottom right: Contrastive VAE architecture with two unimodal encoders and decoders and a contrastive loss for aligning the latent spaces.
• Figure 4: A diffusion models consists of a forward process, which iteratively corrupts an input data sample until it becomes Gaussian noise, and a reverse process, which reconstruct the original data from white noise. Top: Latent diffusion model architecture for unconditional generation. The standard diffusion model architecture is obtained by removing the encoder $p$ and the decoder $q$. Bottom: Conditional multimodal diffusion model with a shared encoder and two unimodal decoders. An unconditional multimodal model is obtained by simply removing the condition on $Y$.
• Figure 5: High-level architecture of an MLLM model. Each input is processed by a specialized encoder to obtain modality specific features which are then projected to a representation adequate for an LLM backbone via suitable adaptors. Similarly, the output of the LLM is then projected to serve as input for specific generators for each modality via a similar adaptor. The input and output modalities are independent.