Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Towards Transfer-Efficient Multi-modal Sequential Recommendation with State Space Duality

Hao Fan, Qingyang Liu, Hongjiu Liu, Yanrong Hu, Kai Fang

Abstract

Sequential Recommendation (SR) models infer user preferences from interaction histories. While transferable Multi-modal SR models outperform traditional ID-based approaches, existing methods struggle with slow fine-tuning convergence due to complex optimization requirements and negative transfer effects. We propose MMM4Rec (Multi-Modal Mamba for Sequential Recommendation), a novel Multi-modal SR framework that incorporates a dedicated algebraic constraint mechanism for efficient transfer learning. By combining State Space Duality (SSD)'s temporal decay properties with a globally-aware temporal modeling design, our model dynamically prioritizes key modality information, overcoming limitations of Transformer-based approaches. The framework implements a constrained two-stage process: (1) sequence-level cross-modal alignment via shared projection matrices, followed by (2) temporal fusion using our newly designed Cross-SSD module and dual-channel Fourier adaptive filtering. This architecture maintains semantic consistency while suppressing noise propagation. MMM4Rec achieves rapid fine-tuning convergence with simple cross-entropy loss, significantly improving Multi-modal recommendation accuracy while maintaining strong transferability. Extensive experiments demonstrate MMM4Rec's state-of-the-art performance, achieving strong multi-modal retrieval capability and exhibiting 10x faster average convergence speed when transferring to large-scale downstream datasets. The implementation is available at https://github.com/AlwaysFHao/MMM4Rec .

Towards Transfer-Efficient Multi-modal Sequential Recommendation with State Space Duality

Abstract

Sequential Recommendation (SR) models infer user preferences from interaction histories. While transferable Multi-modal SR models outperform traditional ID-based approaches, existing methods struggle with slow fine-tuning convergence due to complex optimization requirements and negative transfer effects. We propose MMM4Rec (Multi-Modal Mamba for Sequential Recommendation), a novel Multi-modal SR framework that incorporates a dedicated algebraic constraint mechanism for efficient transfer learning. By combining State Space Duality (SSD)'s temporal decay properties with a globally-aware temporal modeling design, our model dynamically prioritizes key modality information, overcoming limitations of Transformer-based approaches. The framework implements a constrained two-stage process: (1) sequence-level cross-modal alignment via shared projection matrices, followed by (2) temporal fusion using our newly designed Cross-SSD module and dual-channel Fourier adaptive filtering. This architecture maintains semantic consistency while suppressing noise propagation. MMM4Rec achieves rapid fine-tuning convergence with simple cross-entropy loss, significantly improving Multi-modal recommendation accuracy while maintaining strong transferability. Extensive experiments demonstrate MMM4Rec's state-of-the-art performance, achieving strong multi-modal retrieval capability and exhibiting 10x faster average convergence speed when transferring to large-scale downstream datasets. The implementation is available at https://github.com/AlwaysFHao/MMM4Rec .

Paper Structure

This paper contains 33 sections, 25 equations, 8 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Why Mamba Fits SR
  3. Methods
  4. Problem Formulation and Method Overview
  5. Multi-modal Feature Pre-Extraction
  6. Pretrained Multi-modal Encoder
  7. Modality-specific Adapters
  8. Optional Item Modality Bias
  9. Sequence-level Multi-modal Alignment
  10. Time-aware State Space Duality
  11. Modal alignment of SR semantics
  12. Sequential-level Multi-modal Fusion
  13. Dual-Channel Fourier Filtering
  14. Time-aware Cross SSD
  15. Efficient Transfer Training Strategy
  16. ...and 18 more sections

Figures (8)

  • Figure 1: The two main research problems in multi-modal SR: (a) The alignment problem between multi-modal information and recommendation semantic space. (b) The unequal contribution problem of items within interaction sequences.
  • Figure 2: Performance of SASRec at different truncation lengths on the Kindle.
  • Figure 3: The overview of MMM4Rec .
  • Figure 4: Comparison of model convergence speed on Scientific.
  • Figure 5: Comparison of model convergence speed on Pantry.
  • ...and 3 more figures