Self-Supervised Representation Learning with ID-Content Modality Alignment for Sequential Recommendation

TL;DR

SICSRec tackles the dual challenges of data sparsity and modality heterogeneity in sequential recommendation by aligning item ID and content modalities through an LLM-guided data construction process and supervised fine-tuning of text and image encoders. It then learns user preferences with a Transformer-based architecture comprising an ID-modality encoder, a content-modality encoder, and a cross-modality mix-decoder, followed by a content-aware contrastive objective and a LoRA-based efficient fine-tuning scheme. The approach achieves consistent improvements over both ID- and content-based baselines across four NinRec datasets, demonstrating the value of joint modality alignment and two-step training for robust multimodal sequential recommendation. The work offers practical impact for real-world systems by enabling effective multimodal use without prohibitive training costs, and suggests future directions in adaptive fusion and improved cold-start handling.

Abstract

Sequential recommendation (SR) models often capture user preferences based on the historically interacted item IDs, which usually obtain sub-optimal performance when the interaction history is limited. Content-based sequential recommendation has recently emerged as a promising direction that exploits items' textual and visual features to enhance preference learning. However, there are still three key challenges: (i) how to reduce the semantic gap between different content modality representations; (ii) how to jointly model user behavior preferences and content preferences; and (iii) how to design an effective training strategy to align ID representations and content representations. To address these challenges, we propose a novel model, self-supervised representation learning with ID-Content modality alignment, named SICSRec. Firstly, we propose a LLM-driven sample construction method and develop a supervised fine-tuning approach to align item-level modality representations. Secondly, we design a novel Transformer-based sequential model, where an ID-modality sequence encoder captures user behavior preferences, a content-modality sequence encoder learns user content preferences, and a mix-modality sequence decoder grasps the intrinsic relationship between these two types of preferences. Thirdly, we propose a two-step training strategy with a content-aware contrastive learning task to align modality representations and ID representations, which decouples the training process of content modality dependency and item collaborative dependency. Extensive experiments conducted on four public video streaming datasets demonstrate our SICSRec outperforms the state-of-the-art ID-modality sequential recommenders and content-modality sequential recommenders by 8.04% on NDCG@5 and 6.62% on NDCD@10 on average, respectively.

Figures (5)

• Figure 1: The architecture of our SICSRec, including (i) content modality semantic alignment in Section \ref{['item_level_modality_alingment']} and (ii) sequence preference learning in Section \ref{['Sequence-level preference modeling']}. We propose a two-step training strategy with a content-aware contrastive learning task, which is described in Section \ref{['training_strategy']}.
• Figure 2: Supervised fine-tuning analyses. "t2t", "i2i", and "t2i" stand for text-to-text alignment, image-to-image alignment, and text-to-image alignment, respectively. "SFT" refers to the combination of these three alignment tasks.
• Figure 3: Recommendation performance of our SICSRec with different values of the balance parameter $\alpha$, and rank $r$.
• Figure 4: Inference time of SASRec, MISSRec, and our SICSRec.
• Figure 5: Case studies of our SICSRec with different model modifications (left: historical preferred items; right: recommended items).