Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Novel Mamba-based Sequential Recommendation Method

Jun Yuan

TL;DR

Hydra introduces a scalable sequential recommender by employing a multi-head latent Mamba architecture that splits item representations into multiple latent subspaces and interleaves historical context with item information through low-dimensional Mamba blocks. It supports single-domain and multi-domain settings, with the latter enabling a single LLM to be fine-tuned for cross-domain knowledge transfer, dramatically reducing adaptation costs. Empirically, Hydra achieves state-of-the-art performance with far fewer parameters and faster training than Transformer-based baselines and demonstrates practical benefits in online and multi-domain scenarios. Collectively, the approach offers a practical path toward large-scale, knowledge-rich SR systems suitable for real-world deployment.

Abstract

Sequential recommendation (SR), which encodes user activity to predict the next action, has emerged as a widely adopted strategy in developing commercial personalized recommendation systems. Although Transformer-based models have proven effective for sequential recommendation, the complexity of the self-attention module in Transformers scales quadratically with the sequence length. Controlling model complexity is essential for large-scale recommendation systems, as these systems may need to handle billion-scale vocabularies that evolve continuously, as well as user behavior sequences that can exceed tens of thousands in length. In this paper, we propose a novel multi-head latent Mamba architecture, which employs multiple low-dimensional Mamba layers and fully connected layers coupled with positional encoding to simultaneously capture historical and item information within each latent subspace. Our proposed method not only enables scaling up to large-scale parameters but also extends to multi-domain recommendation by integrating and fine-tuning LLMs. Through extensive experiments on public datasets, we demonstrate how Hydra effectively addresses the effectiveness-efficiency dilemma, outperforming state-of-the-art sequential recommendation baselines with significantly fewer parameters and reduced training time.

A Novel Mamba-based Sequential Recommendation Method

TL;DR

Hydra introduces a scalable sequential recommender by employing a multi-head latent Mamba architecture that splits item representations into multiple latent subspaces and interleaves historical context with item information through low-dimensional Mamba blocks. It supports single-domain and multi-domain settings, with the latter enabling a single LLM to be fine-tuned for cross-domain knowledge transfer, dramatically reducing adaptation costs. Empirically, Hydra achieves state-of-the-art performance with far fewer parameters and faster training than Transformer-based baselines and demonstrates practical benefits in online and multi-domain scenarios. Collectively, the approach offers a practical path toward large-scale, knowledge-rich SR systems suitable for real-world deployment.

Abstract

Sequential recommendation (SR), which encodes user activity to predict the next action, has emerged as a widely adopted strategy in developing commercial personalized recommendation systems. Although Transformer-based models have proven effective for sequential recommendation, the complexity of the self-attention module in Transformers scales quadratically with the sequence length. Controlling model complexity is essential for large-scale recommendation systems, as these systems may need to handle billion-scale vocabularies that evolve continuously, as well as user behavior sequences that can exceed tens of thousands in length. In this paper, we propose a novel multi-head latent Mamba architecture, which employs multiple low-dimensional Mamba layers and fully connected layers coupled with positional encoding to simultaneously capture historical and item information within each latent subspace. Our proposed method not only enables scaling up to large-scale parameters but also extends to multi-domain recommendation by integrating and fine-tuning LLMs. Through extensive experiments on public datasets, we demonstrate how Hydra effectively addresses the effectiveness-efficiency dilemma, outperforming state-of-the-art sequential recommendation baselines with significantly fewer parameters and reduced training time.

Paper Structure

This paper contains 31 sections, 17 equations, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Sequential Recommendation
  4. Recommendation with Language Models
  5. Generative Recommendation
  6. Problem Description
  7. Preliminaries
  8. Problem Definition
  9. The Architecture of Hydra
  10. Item Model
  11. Embedding Layer
  12. Item LLM
  13. Hydra Layers
  14. Multi-head Latent Mamba
  15. Feed-Forward Network (FFN)
  16. ...and 16 more sections

Figures (3)

  • Figure 1: Overall Architecture of Hydra. Hydra has an item model, several identical stacked Hydra layers and a prediction layer. The core of Hydra layer is multi-head latent Mamba, which can extract and interact historical and item information to better model long and noisy user context.
  • Figure 2: Two types of Item Model in Hydra. Embedding layer can represent ID features of item. Item LLM can be any pre-trained LLM and encode text into hidden embedding. The gray box marked Hydra is actually the stacked Hydra layers.
  • Figure 3: The architecture of Multi-domain Hydra. Multi-domain Hydra will get two input context sequence, domain-specific and multi-domain user context and predict in each domain. All domains share the stacked Hydra layers.