VIP5: Towards Multimodal Foundation Models for Recommendation

TL;DR

This work tackles the fragmentation across vision, language, and recommender systems by introducing VIP5, a multimodal foundation model built on the P5 paradigm that unifies these modalities through multimodal personalized prompts and adapter-based, parameter-efficient tuning. By freezing the backbone and training lightweight adapters, VIP5 achieves strong performance gains across sequential, direct, and explanation tasks while improving training efficiency and memory usage. Through extensive experiments on four real-world datasets, VIP5 demonstrates the practicality of a single, multimodal interface for recommendation and highlights the value of visual signals in multimodal prompts. The study also outlines limitations and future directions, including fairness considerations, interpretability, scalability to more modalities, and further prompt strategy development.

Abstract

Computer Vision (CV), Natural Language Processing (NLP), and Recommender Systems (RecSys) are three prominent AI applications that have traditionally developed independently, resulting in disparate modeling and engineering methodologies. This has impeded the ability for these fields to directly benefit from each other's advancements. With the recent development of foundation models, large language models have emerged as a potential general-purpose interface for unifying different modalities and problem formulations. In light of this, we propose the development of a multimodal foundation model (MFM) considering visual, textual, and personalization modalities under the P5 recommendation paradigm, thus named VIP5 (Visual P5), to unify various modalities and recommendation tasks. This will enable the processing of multiple modalities in a shared architecture for improved recommendations. To achieve this, we introduce multimodal personalized prompts to accommodate multiple modalities under a shared format. Additionally, we propose a parameter-efficient training method for foundation models, which involves freezing the P5 backbone and fine-tuning lightweight adapters, resulting in improved recommendation performance and increased efficiency in terms of training time and memory usage. Code and data of VIP5 are available at https://github.com/jeykigung/VIP5.

Figures (10)

• Figure 1: An example task scope of VIP5 covering three popular recommendation tasks. Based on multimodal personalized prompts (left) that interleave language and visual tokens, VIP5 is able to transfer all task and all modalities into a unified sequence format, and generates target outputs (right) according to certain task descriptions. VIP5 treats large language models as a fixed general-purpose interface and finetunes extra visual and language processing layers to achieve the ability for handling various recommendation tasks.
• Figure 2: An illustration of the VIP5 framework. VIP5 is built on an encoder--decoder Transformer model that takes in textual inputs as well as image inputs to produce responses or make recommendation decisions. In the figure, a fire symbol represents training with parameter update while a snow flake symbol stands for the frozen parameters.
• Figure 3: Performance comparison between text-based prompt and multimodal prompt.
• Figure 4: Performance comparison among only activating adapters in self-attention blocks, both self-attention and cross-attention blocks, and full finetuning.
• Figure 5: Ablation on the downsample reduction rate.