HyMiRec: A Hybrid Multi-interest Learning Framework for LLM-based Sequential Recommendation

TL;DR

HyMiRec tackles the challenge of modeling long-term and diverse user interests in sequential recommendation by coupling a lightweight recommender for coarse, long-history signals with an LLM-based recommender for refined, short-term signals. It introduces a cosine-similarity-based residual codebook to efficiently compress and reuse historical embeddings, and a Disentangled Multi-Interest Learning module that uses window-based targets and a Hungarian-matching driven contrastive loss to learn multiple distinct interests. The approach demonstrates superior offline performance on public benchmarks and a large industrial dataset, and yields tangible gains in online production across item cold-start and ad cold-start scenarios. Overall, HyMiRec offers a scalable, effective solution for long-tail and diverse user preferences in real-world recommender systems, with clear practical impact and avenues for future enhancement.

Abstract

Large language models (LLMs) have recently demonstrated strong potential for sequential recommendation. However, current LLM-based approaches face critical limitations in modeling users' long-term and diverse interests. First, due to inference latency and feature fetching bandwidth constraints, existing methods typically truncate user behavior sequences to include only the most recent interactions, resulting in the loss of valuable long-range preference signals. Second, most current methods rely on next-item prediction with a single predicted embedding, overlooking the multifaceted nature of user interests and limiting recommendation diversity. To address these challenges, we propose HyMiRec, a hybrid multi-interest sequential recommendation framework, which leverages a lightweight recommender to extracts coarse interest embeddings from long user sequences and an LLM-based recommender to captures refined interest embeddings. To alleviate the overhead of fetching features, we introduce a residual codebook based on cosine similarity, enabling efficient compression and reuse of user history embeddings. To model the diverse preferences of users, we design a disentangled multi-interest learning module, which leverages multiple interest queries to learn disentangles multiple interest signals adaptively, allowing the model to capture different facets of user intent. Extensive experiments are conducted on both benchmark datasets and a collected industrial dataset, demonstrating our effectiveness over existing state-of-the-art methods. Furthermore, online A/B testing shows that HyMiRec brings consistent improvements in real-world recommendation systems. Code is available at https://github.com/FireRedTeam/FireRedSeqRec.

Figures (3)

• Figure 1: Comparison of our method and existing methods. A user often has multiple distinct interests, also including both long-term and short-term preferences. (a) Existing methods: truncate history sequences and use a single embedding to learn multiple interests, which may cause long-term interest forgetting and entangled representations. (b) Our method: adopts a hybrid framework to model long- and short-term behaviors with multi-interest embeddings, enabling effective learning of user's multiple interests.
• Figure 2: Overview of the proposed HyMiRec. In Stage 1, a content LLM encoder and an LLM recommender are jointly trained on short sequences, where the encoder extracts item embeddings and the recommender predicts user interests. The trained encoder is then fixed to build a cosine-similarity-based residual codebook and compress item embeddings into quantization codes. In Stage 2, long user histories are reconstructed from codes and processed by a lightweight recommender with learnable queries to yield coarse interests, which are combined with recent interactions and refined by the LLM recommender. The final interest embeddings are optimized with the proposed Disentangled Multi-Interest Learning (DMIL) module using future window targets.
• Figure 3: Performance of our HyMiRec under different hyper-parameters.