Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Unleashing the Potential of Sparse Attention on Long-term Behaviors for CTR Prediction

Weijiang Lai, Beihong Jin, Di Zhang, Siru Chen, Jiongyan Zhang, Yuhang Gou, Jian Dong, Xingxing Wang

TL;DR

SparseCTR tackles the challenge of modeling long-term user behaviors in CTR prediction by introducing TimeChunking-based personalized segmentation, a three-branch EvoAttention mechanism (Global, Transition, Local), and a composite RelTemporal encoding to inject temporal signals into attention. The approach achieves both high efficiency and strong predictive performance, exhibiting clear scaling laws across three orders of FLOPs and enabling deployment with longer sequences and larger models. Offline evaluations on three real-world datasets and online A/B tests demonstrate state-of-the-art results and tangible business gains (CTR and CPM) with modest latency. The work provides practical techniques for scalable, personalized long-term user modeling in industrial recommender systems and offers code for replication.

Abstract

In recent years, the success of large language models (LLMs) has driven the exploration of scaling laws in recommender systems. However, models that demonstrate scaling laws are actually challenging to deploy in industrial settings for modeling long sequences of user behaviors, due to the high computational complexity of the standard self-attention mechanism. Despite various sparse self-attention mechanisms proposed in other fields, they are not fully suited for recommendation scenarios. This is because user behaviors exhibit personalization and temporal characteristics: different users have distinct behavior patterns, and these patterns change over time, with data from these users differing significantly from data in other fields in terms of distribution. To address these challenges, we propose SparseCTR, an efficient and effective model specifically designed for long-term behaviors of users. To be precise, we first segment behavior sequences into chunks in a personalized manner to avoid separating continuous behaviors and enable parallel processing of sequences. Based on these chunks, we propose a three-branch sparse self-attention mechanism to jointly identify users' global interests, interest transitions, and short-term interests. Furthermore, we design a composite relative temporal encoding via learnable, head-specific bias coefficients, better capturing sequential and periodic relationships among user behaviors. Extensive experimental results show that SparseCTR not only improves efficiency but also outperforms state-of-the-art methods. More importantly, it exhibits an obvious scaling law phenomenon, maintaining performance improvements across three orders of magnitude in FLOPs. In online A/B testing, SparseCTR increased CTR by 1.72\% and CPM by 1.41\%. Our source code is available at https://github.com/laiweijiang/SparseCTR.

Unleashing the Potential of Sparse Attention on Long-term Behaviors for CTR Prediction

TL;DR

SparseCTR tackles the challenge of modeling long-term user behaviors in CTR prediction by introducing TimeChunking-based personalized segmentation, a three-branch EvoAttention mechanism (Global, Transition, Local), and a composite RelTemporal encoding to inject temporal signals into attention. The approach achieves both high efficiency and strong predictive performance, exhibiting clear scaling laws across three orders of FLOPs and enabling deployment with longer sequences and larger models. Offline evaluations on three real-world datasets and online A/B tests demonstrate state-of-the-art results and tangible business gains (CTR and CPM) with modest latency. The work provides practical techniques for scalable, personalized long-term user modeling in industrial recommender systems and offers code for replication.

Abstract

In recent years, the success of large language models (LLMs) has driven the exploration of scaling laws in recommender systems. However, models that demonstrate scaling laws are actually challenging to deploy in industrial settings for modeling long sequences of user behaviors, due to the high computational complexity of the standard self-attention mechanism. Despite various sparse self-attention mechanisms proposed in other fields, they are not fully suited for recommendation scenarios. This is because user behaviors exhibit personalization and temporal characteristics: different users have distinct behavior patterns, and these patterns change over time, with data from these users differing significantly from data in other fields in terms of distribution. To address these challenges, we propose SparseCTR, an efficient and effective model specifically designed for long-term behaviors of users. To be precise, we first segment behavior sequences into chunks in a personalized manner to avoid separating continuous behaviors and enable parallel processing of sequences. Based on these chunks, we propose a three-branch sparse self-attention mechanism to jointly identify users' global interests, interest transitions, and short-term interests. Furthermore, we design a composite relative temporal encoding via learnable, head-specific bias coefficients, better capturing sequential and periodic relationships among user behaviors. Extensive experimental results show that SparseCTR not only improves efficiency but also outperforms state-of-the-art methods. More importantly, it exhibits an obvious scaling law phenomenon, maintaining performance improvements across three orders of magnitude in FLOPs. In online A/B testing, SparseCTR increased CTR by 1.72\% and CPM by 1.41\%. Our source code is available at https://github.com/laiweijiang/SparseCTR.
Paper Structure (40 sections, 18 equations, 4 figures, 3 tables)

This paper contains 40 sections, 18 equations, 4 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Long-term Behavior Modeling
  4. Sparse Self-attention Mechanisms
  5. Relative Information Encoding
  6. Methodology
  7. Problem Formulation
  8. Input Layer
  9. SparseBlock
  10. Evolutionary Sparse Self-attention
  11. Personalized Time-aware Chunking
  12. Global Attention
  13. Transition Attention
  14. Local Attention
  15. Sparse Self-attention Fusion
  16. ...and 25 more sections

Figures (4)

  • Figure 1: Architecture of SparseCTR.
  • Figure 2: Comparison with existing sparse self-attention, chunking, and relative encoding methods.
  • Figure 3: Performance of EvoAttention under different configurations in terms of AUC, speed, and GPU memory.
  • Figure 4: Scaling laws on the industrial dataset: AUC under different FLOPs, with FLOPs determined by sequence length and model scale (#Params is the number of non-embedding parameters).