Efficient Sequential Recommendation for Long Term User Interest Via Personalization

TL;DR

Long sequences in sequential recommendation trigger quadratic transformer costs. The authors propose personalized experts that compress long histories into learnable tokens, which are then combined with recent interactions to produce recommendations. The approach works across multiple architectures (HSTU, HLLM) and shows near full-sequence performance on MerRec and EB-NeRD while reducing inference cost. This framework advances scalable, accurate sequential recommendation and is complemented by analyses of training, inference, and token placement, with open-source code for reproducibility.

Abstract

Recent years have witnessed success of sequential modeling, generative recommender, and large language model for recommendation. Though the scaling law has been validated for sequential models, it showed inefficiency in computational capacity when considering real-world applications like recommendation, due to the non-linear(quadratic) increasing nature of the transformer model. To improve the efficiency of the sequential model, we introduced a novel approach to sequential recommendation that leverages personalization techniques to enhance efficiency and performance. Our method compresses long user interaction histories into learnable tokens, which are then combined with recent interactions to generate recommendations. This approach significantly reduces computational costs while maintaining high recommendation accuracy. Our method could be applied to existing transformer based recommendation models, e.g., HSTU and HLLM. Extensive experiments on multiple sequential models demonstrate its versatility and effectiveness. Source code is available at \href{https://github.com/facebookresearch/PerSRec}{https://github.com/facebookresearch/PerSRec}.

Figures (10)

• Figure 1: Performance (Recall@5) of HSTU and HLLM model steadily improves as the sequence length grows from $128$ to $2000$ on MerRec dataset. HLLM could achieves good performance even with short sequence and HSTU reduces the gap as the sequence get longer.
• Figure 2: An overview of the architecture of proposed method. The model divides the long sequence into multiple segments and utilizes a segment decoder to "compress" each segments into segment embedding(s). Those segment embeddings are then combined with item embedding from the most recent segments to perform the sequential recommendation. This design could significantly improve the efficiency. The decoder and segment decoder could share the parameter.
• Figure 3: Each item $x$ and the learnable token $y$ could attention to itself, its preceding items in the same segments, all learnable tokens from the previous segments. The yellow indicates the positions (learnable tokens) which masked off for computing the loss during training. Note the arrows indicates token attend to other positions are not shown here to avoid the figure being overcrowded.
• Figure 4: Illustration of the attention mask. Row i Column j being yellow indicates Position i could attend to Position j. Left: ordinary causal mask used by HSTU and HLLM for next item prediction; Right: modified attention mask to stop item of one segment attending to other segments. Here we use an UIH with four segments as example, the length of each segment is $[8, 12, 8, 16]$ accordingly and after each segment (except the last one) we append one learnable tokens.
• Figure 5: For inference, we first generate and save the activations of the learnable tokens for each segment; then those activations served as KV cache and applied to new item prediction.