A Remarkably Efficient Paradigm to Multimodal Large Language Models for Sequential Recommendation

TL;DR

This paper tackles the efficiency and effectiveness gaps of multimodal LLM-based sequential recommendations by introducing Speeder, a paradigm that combines Multimodal Representation Compression (MRC), Modality-aware Progressive Optimization (MPO), and Sequential Position Awareness Enhancement (SPAE). By compressing multimodal item representations with Mixture of Modality Experts and guiding the LLM through relative and absolute position cues, Speeder achieves strong recommendation performance while dramatically reducing training and inference costs. The approach is validated on three real-world Amazon datasets, where Speeder outperforms state-of-the-art baselines and shows substantial speedups (training up to 2.5× and inference up to 4×) over competing MLLM-based SR models. The work demonstrates that careful modality integration and sequence-aware prompting can yield both practical efficiency and improved long-sequence recommendations, with broad implications for scalable, multimodal user modeling.

Abstract

Sequential recommendations (SR) predict users' future interactions based on their historical behavior. The rise of Large Language Models (LLMs) has brought powerful generative and reasoning capabilities, significantly enhancing SR performance, while Multimodal LLMs (MLLMs) further extend this by introducing data like images and interactive relationships. However, critical issues remain, i.e., (a) Suboptimal item representations caused by lengthy and redundant descriptions, leading to inefficiencies in both training and inference; (b) Modality-related cognitive bias, as LLMs are predominantly pretrained on textual data, limiting their ability to effectively integrate and utilize non-textual modalities; (c) Weakening sequential perception in long interaction sequences, where attention mechanisms struggle to capture earlier interactions, hindering the modeling of long-range dependencies. To address these issues, we propose Speeder, an efficient MLLM-based paradigm for SR featuring three key innovations: 1) Multimodal Representation Compression (MRC), which condenses item attributes into concise yet informative tokens, reducing redundancy and computational cost; 2) Modality-aware Progressive Optimization (MPO), enabling gradual learning of multimodal representations; 3) Sequential Position Awareness Enhancement (SPAE), improving the LLM's capability to capture both relative and absolute sequential dependencies in long interaction sequences. Extensive experiments on real-world datasets demonstrate the effectiveness and efficiency of Speeder. Speeder increases training speed to 250% of the original while reducing inference time to 25% on the Amazon dataset.

Figures (8)

• Figure 1: Motivation of Speeder. Issue 1 reveals suboptimal item representaions in LLM, Issue 2 implies the modality-related cognitive bias, and Issue 3 illustrates the weakening of sequential position awareness facing long sequence. Different colors in the prompt represent distinct items, while non-textual embeddings are generated from an encoder using non-textual data.
• Figure 2: The framework of Speeder. Speeder contains three modules. First, MRC condenses item's attributes into a compact but informative token. Second, SPAE augments LLM's ability of capturing the relative and absolute order in the interaction sequence. Third, MPO allows Speeder to gradually understand and utilize multimodal representation. MHSA, T-FFN, V-FFN, S-FFN, MM-FFN are short for Multi-Head Self-Attention, Textual-FFN, Visual-FFN, Sequential-FFN, Multimodal-FFN, respectively.
• Figure 3: The framework of Hybrid Prompt.
• Figure 4: The pipeline of MPO. (a) In the first stage, only the Textual-FFN is trainable, using text-only data. (b) In the second stage, solely the Sequential-FFN is frozen, with training conducted on paired text and vision data. (c) In the third stage, all parameters are jointly optimized on multimodal data. Trainable parameters in the same block at different stages are shared.
• Figure 5: Performance of Position Proxy Task.