Hierarchical Time-Aware Mixture of Experts for Multi-Modal Sequential Recommendation

TL;DR

This work tackles multi-modal sequential recommendation by addressing redundant content and underutilized explicit temporal signals. It proposes HM4SR, a hierarchical time-aware MoE framework with Interactive MoE for cross-modal feature extraction and Temporal MoE for explicit time encoding, complemented by a Transformer-based user learner. The model is trained with a multi-task objective that includes sequence-level category prediction, ID-based contrastive learning, and placeholder contrastive learning to mitigate data sparsity and enhance modality-time integration. Experiments on four public datasets demonstrate state-of-the-art performance and underline the practical impact of explicit temporal information combined with rich multi-modal item representations for capturing dynamic user preferences.

Abstract

Multi-modal sequential recommendation (SR) leverages multi-modal data to learn more comprehensive item features and user preferences than traditional SR methods, which has become a critical topic in both academia and industry. Existing methods typically focus on enhancing multi-modal information utility through adaptive modality fusion to capture the evolving of user preference from user-item interaction sequences. However, most of them overlook the interference caused by redundant interest-irrelevant information contained in rich multi-modal data. Additionally, they primarily rely on implicit temporal information based solely on chronological ordering, neglecting explicit temporal signals that could more effectively represent dynamic user interest over time. To address these limitations, we propose a Hierarchical time-aware Mixture of experts for multi-modal Sequential Recommendation (HM4SR) with a two-level Mixture of Experts (MoE) and a multi-task learning strategy. Specifically, the first MoE, named Interactive MoE, extracts essential user interest-related information from the multi-modal data of each item. Then, the second MoE, termed Temporal MoE, captures user dynamic interests by introducing explicit temporal embeddings from timestamps in modality encoding. To further address data sparsity, we propose three auxiliary supervision tasks: sequence-level category prediction (CP) for item feature understanding, contrastive learning on ID (IDCL) to align sequence context with user interests, and placeholder contrastive learning (PCL) to integrate temporal information with modalities for dynamic interest modeling. Extensive experiments on four public datasets verify the effectiveness of HM4SR compared to several state-of-the-art approaches.

Figures (8)

• Figure 1: A toy example of user-item interaction sequence.
• Figure 2: The overall architecture of the proposed HM4SR.
• Figure 3: Performance comparison of removing each designed component or each modality.
• Figure 4: Performance comparison of HM4SR w.r.t different hyper-parameters.
• Figure 5: Performance comparison on different time embeddings inputted to the gating router of Temporal MoE.