Revisiting scalable sequential recommendation with Multi-Embedding Approach and Mixture-of-Experts
Qiushi Pan, Hao Wang, Guoyuan An, Luankang Zhang, Wei Guo, Yong Liu
TL;DR
The paper tackles scalable sequential recommendation by addressing the representational bottleneck of a single item embedding. It introduces Fuxi-MME, which combines a multi-embedding strategy with a sparsely-gated Mixture-of-Experts (MoE) layer on the Fuxi-α backbone to enable adaptive, context-aware processing of diverse item attributes. The method achieves state-of-the-art results on three public datasets, with ablations confirming the complementary benefits of disentangled embeddings and expert routing. This work demonstrates that principled structural representation and adaptive computation can yield substantial gains in scalable SR and opens avenues for interpretable, multi-modal extensions.
Abstract
In recommendation systems, how to effectively scale up recommendation models has been an essential research topic. While significant progress has been made in developing advanced and scalable architectures for sequential recommendation(SR) models, there are still challenges due to items' multi-faceted characteristics and dynamic item relevance in the user context. To address these issues, we propose Fuxi-MME, a framework that integrates a multi-embedding strategy with a Mixture-of-Experts (MoE) architecture. Specifically, to efficiently capture diverse item characteristics in a decoupled manner, we decompose the conventional single embedding matrix into several lower-dimensional embedding matrices. Additionally, by substituting relevant parameters in the Fuxi Block with an MoE layer, our model achieves adaptive and specialized transformation of the enriched representations. Empirical results on public datasets show that our proposed framework outperforms several competitive baselines.