VLM4Rec: Multimodal Semantic Representation for Recommendation with Large Vision-Language Models

Abstract

Multimodal recommendation is commonly framed as a feature fusion problem, where textual and visual signals are combined to better model user preference. However, the effectiveness of multimodal recommendation may depend not only on how modalities are fused, but also on whether item content is represented in a semantic space aligned with preference matching. This issue is particularly important because raw visual features often preserve appearance similarity, while user decisions are typically driven by higher-level semantic factors such as style, material, and usage context. Motivated by this observation, we propose LVLM-grounded Multimodal Semantic Representation for Recommendation (VLM4Rec), a lightweight framework that organizes multimodal item content through semantic alignment rather than direct feature fusion. VLM4Rec first uses a large vision-language model to ground each item image into an explicit natural-language description, and then encodes the grounded semantics into dense item representations for preference-oriented retrieval. Recommendation is subsequently performed through a simple profile-based semantic matching mechanism over historical item embeddings, yielding a practical offline-online decomposition. Extensive experiments on multiple multimodal recommendation datasets show that VLM4Rec consistently improves performance over raw visual features and several fusion-based alternatives, suggesting that representation quality may matter more than fusion complexity in this setting. The code is released at https://github.com/tyvalencia/enhancing-mm-rec-sys.

Figures (4)

• Figure 1: Overview of the VLM4Rec framework. Multimodal item content (images, titles, and user--item interactions) is processed through multiple representation paths: the proposed LVLM grounding path (top), which transforms item images into rich semantic descriptions via LLaVA-NeXT 7B and encodes them with Sentence-BERT. Baseline fusion and learned combination methods including concatenation, gating, and attention (middle), and SMORE spectral fusion (top). All embedding variants feed into the same lightweight retrieval stage, where user profiles are constructed via mean pooling and candidates are ranked by cosine similarity.
• Figure 2: Full test set comparison with uncertainty. Horizontal bars show mean Recall@10 and NDCG@10 performance across all seven models evaluated on the complete dataset (23,318 users); error bars denote user-level standard deviation. BERT text-only remains strongest on the full test set, while attention fusion is the closest multimodal alternative.
• Figure 3: LLaVA-covered subset metric heatmap. Rows grouped by model type; columns show Recall@5, @10, @20, NDCG@10, @20, and Hit@10. Color intensity indicates performance (higher is better). LLaVA-based representations consistently outperform BERT-based variants across all metrics.
• Figure 4: Modality contribution profile across metrics. Line plot showing performance trajectories for text-only, vision-only, naive fusion, and learned fusion strategies. Across all metrics, text-only remains strongest; learned fusion tracks closely and clearly improves over naive fusion, while vision-only lags throughout.