Harnessing Multimodal Large Language Models for Multimodal Sequential Recommendation

TL;DR

This work addresses multimodal sequential recommendation by leveraging Multimodal Large Language Models (MLLMs) to handle both visual and textual item data. It introduces MLLM-MSR, a two-stage approach that first converts multimodal item content into unified textual descriptions and then uses a recurrent, prompt-driven scheme to infer dynamic user preferences, followed by supervised fine-tuning (SFT) of an MLLM-based recommender using LoRA for efficiency. Key contributions include (i) a novel multimodal item summarization workflow, (ii) recurrent, prompt-based inference of evolving user preferences, and (iii) end-to-end fine-tuning of an MLLM recommender that demonstrates superior performance and interpretability across three real-world datasets. The results suggest that integrating multimodal data with structured prompt design and efficient fine-tuning yields significant improvements in personalization and stability when modeling evolving user interests, with practical impact for scalable, multimodal RS deployments. The approach is grounded in a loss $L = -\sum_{i=1}^L \log P(v_i|v_{<i}, \mathcal{I})$ and a token-based decision rule $p = \frac{p(\text{'yes'})}{p(\text{'yes'}) + p(\text{'no'})}$, illustrating the probabilistic framing behind both training and prediction.

Abstract

Recent advances in Large Language Models (LLMs) have demonstrated significant potential in the field of Recommendation Systems (RSs). Most existing studies have focused on converting user behavior logs into textual prompts and leveraging techniques such as prompt tuning to enable LLMs for recommendation tasks. Meanwhile, research interest has recently grown in multimodal recommendation systems that integrate data from images, text, and other sources using modality fusion techniques. This introduces new challenges to the existing LLM-based recommendation paradigm which relies solely on text modality information. Moreover, although Multimodal Large Language Models (MLLMs) capable of processing multi-modal inputs have emerged, how to equip MLLMs with multi-modal recommendation capabilities remains largely unexplored. To this end, in this paper, we propose the Multimodal Large Language Model-enhanced Multimodaln Sequential Recommendation (MLLM-MSR) model. To capture the dynamic user preference, we design a two-stage user preference summarization method. Specifically, we first utilize an MLLM-based item-summarizer to extract image feature given an item and convert the image into text. Then, we employ a recurrent user preference summarization generation paradigm to capture the dynamic changes in user preferences based on an LLM-based user-summarizer. Finally, to enable the MLLM for multi-modal recommendation task, we propose to fine-tune a MLLM-based recommender using Supervised Fine-Tuning (SFT) techniques. Extensive evaluations across various datasets validate the effectiveness of MLLM-MSR, showcasing its superior ability to capture and adapt to the evolving dynamics of user preferences.

Figures (6)

• Figure 1: The schematic framework of MLLM-MSR
• Figure 2: An example of recurrent user preference inference.
• Figure 3: An example of MLLM-based sequential recommendation
• Figure 4: The performance of MLLM-MSR and its variants.
• Figure 5: Performance of MLLM-MSR under different block size.

Theorems & Definitions (1)

• Definition 1: Multimodal Sequential Recommendation