VLM2Rec: Resolving Modality Collapse in Vision-Language Model Embedders for Multimodal Sequential Recommendation

Abstract

Sequential Recommendation (SR) in multimodal settings typically relies on small frozen pretrained encoders, which limits semantic capacity and prevents Collaborative Filtering (CF) signals from being fully integrated into item representations. Inspired by the recent success of Large Language Models (LLMs) as high-capacity embedders, we investigate the use of Vision-Language Models (VLMs) as CF-aware multimodal encoders for SR. However, we find that standard contrastive supervised fine-tuning (SFT), which adapts VLMs for embedding generation and injects CF signals, can amplify its inherent modality collapse. In this state, optimization is dominated by a single modality while the other degrades, ultimately undermining recommendation accuracy. To address this, we propose VLM2Rec, a VLM embedder-based framework for multimodal sequential recommendation designed to ensure balanced modality utilization. Specifically, we introduce Weak-modality Penalized Contrastive Learning to rectify gradient imbalance during optimization and Cross-Modal Relational Topology Regularization to preserve geometric consistency between modalities. Extensive experiments demonstrate that VLM2Rec consistently outperforms state-of-the-art baselines in both accuracy and robustness across diverse scenarios.

Figures (5)

• Figure 1: Analysis of modality collapse via dropout test and gradient dynamics. SFT makes the image modality act as a negative transfer when fused with text, because of the overlooked gradient signal of the weak modality during training. Our proposed VLM2Rec successfully re-balances modality gradients, enabling stable multimodal gains.
• Figure 2: Left: Our framework encodes text/image sequences/items to enable two usages: Task 1) direct sequence–item recommendation and Task 2) VLM2Rec-generated item embedding initialization for downstream SR models. Right: We fine-tune the VLM with $\mathcal{L}_{\text{WPCL}}$ to adaptively penalize the user-specific weak modality (restoring negative separation) and $\mathcal{L}_{\text{CRTR}}$ to align cross-modal relational topology, preventing geometric distortion while preserving modality individuality.
• Figure 3: Cold-start evaluation on the Beauty for Task 2 (SASRec), grouped by target item frequency in the training set.
• Figure 4: Performance of VLM2Rec across various VLM families and parameter sizes, reporting N@20 for Task 1 and Task 2.
• Figure 5: Impact of hyperparameters $\tau_{\text{WPCL}},\tau_{\text{CRTR}}$, and $\lambda$.