Long-Sequence Recommendation Models Need Decoupled Embeddings

TL;DR

This work identifies a fundamental interference problem in long-sequence recommendations: shared embeddings for attention and representation cause gradient domination and conflicts, degrading both attention accuracy and representation discriminability. It introduces DARE, which uses two independent embedding tables for attention and representation, enabling full decoupling and superior optimization of each task. Offline results show consistent AUC gains up to approximately $0.9\%$ on public datasets, while online deployment on Tencent’s platform yields a $1.47\%$ GMV lift and notable cost reductions. Additionally, decoupling allows reducing the attention embedding dimension to accelerate search by about $50\%$ with limited performance loss, illustrating practical benefits for real-time serving. Overall, DARE advances long-sequence recommendation by improving retrieval quality, discriminability, and serving efficiency, with strong empirical support across datasets and a clear path to scalable deployment.

Abstract

Lifelong user behavior sequences are crucial for capturing user interests and predicting user responses in modern recommendation systems. A two-stage paradigm is typically adopted to handle these long sequences: a subset of relevant behaviors is first searched from the original long sequences via an attention mechanism in the first stage and then aggregated with the target item to construct a discriminative representation for prediction in the second stage. In this work, we identify and characterize, for the first time, a neglected deficiency in existing long-sequence recommendation models: a single set of embeddings struggles with learning both attention and representation, leading to interference between these two processes. Initial attempts to address this issue with some common methods (e.g., linear projections -- a technique borrowed from language processing) proved ineffective, shedding light on the unique challenges of recommendation models. To overcome this, we propose the Decoupled Attention and Representation Embeddings (DARE) model, where two distinct embedding tables are initialized and learned separately to fully decouple attention and representation. Extensive experiments and analysis demonstrate that DARE provides more accurate searches of correlated behaviors and outperforms baselines with AUC gains up to 0.9% on public datasets and notable improvements on Tencent's advertising platform. Furthermore, decoupling embedding spaces allows us to reduce the attention embedding dimension and accelerate the search procedure by 50% without significant performance impact, enabling more efficient, high-performance online serving. Code in PyTorch for experiments, including model analysis, is available at https://github.com/thuml/DARE.

Figures (20)

• Figure 1: Overview of our work. During search, only a limited number of important behaviors are retrieved according to their attention scores. During sequence modeling, the selected behaviors are aggregated into a discriminative representation for prediction. Our DARE model decouples the embeddings used in attention calculation and representation aggregation, effectively resolving their conflict and leading to improved performance and faster inference speed.
• Figure 2: The magnitude of embedding gradients from the attention and representation modules.
• Figure 3: Cosine angles of gradients.
• Figure 4: Illustration and evaluation for adopting linear projections. (a-b) The attention module in the original TWIN and after adopting linear projections. (c) Performance of TWIN variants. Adopting linear projections causes an AUC drop of nearly 2% on Taobao.
• Figure 5: The influence of linear projections with different embedding dimensions in NLP.