Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rec-GPT4V: Multimodal Recommendation with Large Vision-Language Models

Yuqing Liu, Yu Wang, Lichao Sun, Philip S. Yu

TL;DR

This work tackles the challenge of employing large vision-language models (LVLMs) for multimodal recommendations, focusing on LVLMs' lack of user-specific guidance and their difficulty with multiple, noisy images. It introduces Visual-Summary Thought (VST), a prompting strategy that converts per-image content into textual summaries and fuses them with user history to rerank candidate items. Across four datasets and three LVLM backbones (GPT4-V, LLaVA-7b, LLaVA-13b), VST consistently improves ranking performance over baselines that rely on simple concatenation or standard in-context strategies, with notable gains in domains where title noise is high. The results demonstrate that distilling image information into targeted textual summaries can better leverage LVLMs for personalized recommendations and offer a practical path toward more effective LVLM-based multimodal recommender systems.

Abstract

The development of large vision-language models (LVLMs) offers the potential to address challenges faced by traditional multimodal recommendations thanks to their proficient understanding of static images and textual dynamics. However, the application of LVLMs in this field is still limited due to the following complexities: First, LVLMs lack user preference knowledge as they are trained from vast general datasets. Second, LVLMs suffer setbacks in addressing multiple image dynamics in scenarios involving discrete, noisy, and redundant image sequences. To overcome these issues, we propose the novel reasoning scheme named Rec-GPT4V: Visual-Summary Thought (VST) of leveraging large vision-language models for multimodal recommendation. We utilize user history as in-context user preferences to address the first challenge. Next, we prompt LVLMs to generate item image summaries and utilize image comprehension in natural language space combined with item titles to query the user preferences over candidate items. We conduct comprehensive experiments across four datasets with three LVLMs: GPT4-V, LLaVa-7b, and LLaVa-13b. The numerical results indicate the efficacy of VST.

Rec-GPT4V: Multimodal Recommendation with Large Vision-Language Models

TL;DR

This work tackles the challenge of employing large vision-language models (LVLMs) for multimodal recommendations, focusing on LVLMs' lack of user-specific guidance and their difficulty with multiple, noisy images. It introduces Visual-Summary Thought (VST), a prompting strategy that converts per-image content into textual summaries and fuses them with user history to rerank candidate items. Across four datasets and three LVLM backbones (GPT4-V, LLaVA-7b, LLaVA-13b), VST consistently improves ranking performance over baselines that rely on simple concatenation or standard in-context strategies, with notable gains in domains where title noise is high. The results demonstrate that distilling image information into targeted textual summaries can better leverage LVLMs for personalized recommendations and offer a practical path toward more effective LVLM-based multimodal recommender systems.

Abstract

The development of large vision-language models (LVLMs) offers the potential to address challenges faced by traditional multimodal recommendations thanks to their proficient understanding of static images and textual dynamics. However, the application of LVLMs in this field is still limited due to the following complexities: First, LVLMs lack user preference knowledge as they are trained from vast general datasets. Second, LVLMs suffer setbacks in addressing multiple image dynamics in scenarios involving discrete, noisy, and redundant image sequences. To overcome these issues, we propose the novel reasoning scheme named Rec-GPT4V: Visual-Summary Thought (VST) of leveraging large vision-language models for multimodal recommendation. We utilize user history as in-context user preferences to address the first challenge. Next, we prompt LVLMs to generate item image summaries and utilize image comprehension in natural language space combined with item titles to query the user preferences over candidate items. We conduct comprehensive experiments across four datasets with three LVLMs: GPT4-V, LLaVa-7b, and LLaVa-13b. The numerical results indicate the efficacy of VST.
Paper Structure (17 sections, 4 figures, 2 tables)

This paper contains 17 sections, 4 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Problem definition
  4. Preliminary
  5. LVLMs exhibit limitations in handling multiple images
  6. Visual-Summary Generation
  7. Visual-Summary Thought for MMRSs
  8. Experiments
  9. Experimental Settings
  10. Dataset
  11. Metrics
  12. Implementation Details
  13. Baseline Models
  14. Overall Performance
  15. Ablation Study
  16. ...and 2 more sections

Figures (4)

  • Figure 1: Title-only vs title-image concatenation performance comparison on GPT4-V.
  • Figure 2: Framework of Visual-Summary Thought of LVLMs for Multimodal Recommendation
  • Figure 3: Ablation study. Performance of LLaVA-13b with different prompts on Toys dataset.
  • Figure 4: Case study. Text in red indicates the target item. Text in orange, purple, or blue indicates the pattern to describe the item for the corresponding prompt. Text in yellow highlights some key features obtained through visual-summary generation.