GRAB: An LLM-Inspired Sequence-First Click-Through Rate Prediction Modeling Paradigm

TL;DR

GRAB reframes CTR prediction as end-to-end generative ranking by integrating DLRM-style sparse features with autoregressive sequence modeling. The core innovations—CamA for action-aware, multi-channel attention and the STS training strategy—address long-sequence modeling, heterogeneous user actions, and training stability in industrial settings. Offline and online evaluations show consistent gains over strong baselines, with GRAB achieving notable improvements in AUC, CTR, and CPM and displaying scalable behavior when sequence length and model capacity grow. The deployment demonstrates practical viability in a compute-bound, real-time ad system, highlighting GRAB's potential to redefine large-scale CTR modeling and open avenues for multimodal and universal user representations.

Abstract

Traditional Deep Learning Recommendation Models (DLRMs) face increasing bottlenecks in performance and efficiency, often struggling with generalization and long-sequence modeling. Inspired by the scaling success of Large Language Models (LLMs), we propose Generative Ranking for Ads at Baidu (GRAB), an end-to-end generative framework for Click-Through Rate (CTR) prediction. GRAB integrates a novel Causal Action-aware Multi-channel Attention (CamA) mechanism to effectively capture temporal dynamics and specific action signals within user behavior sequences. Full-scale online deployment demonstrates that GRAB significantly outperforms established DLRMs, delivering a 3.05% increase in revenue and a 3.49% rise in CTR. Furthermore, the model demonstrates desirable scaling behavior: its expressive power shows a monotonic and approximately linear improvement as longer interaction sequences are utilized.

Figures (8)

• Figure 1: The traditional DLRM architecture: sparse features are hashed to IDs and embedded via PSTable, and then concatenated into a fixed-length flattened vector for CTR prediction.
• Figure 2: Overview of GRAB’s end-to-end CTR prediction pipeline: (1) Tokenizing raw fields via a sparse PSTable and fusing them into event tokens. (2) Packing tokens per user with causal and heterogeneous masks. (3) Processing through $N$ Transformer layers equipped with the Causal Action-aware Multi-channel Attention (CamA) mechanism. (4) Final CTR prediction from the output representations.
• Figure 3: Sequence packing and user-isolated causal masking in GRAB. (a) Instead of padding each impression instance to a fixed length $L_{\max}$, tokens from multiple impressions are concatenated within each user and different users are kept in disjoint segments, yielding a single packed sequence of length $N_{token}$ for compute-efficient batching. (b) The user-isolated causal mask exhibits a block-diagonal lower-triangular pattern, so each token can only attend to past tokens within the same user segment, enforcing both user isolation and temporal causality.
• Figure 4: Heterogeneous behavior tokens and heterogeneous visibility mask $M^{\mathrm{het}}$ (blue entries). Partial tokens attend only to partial-history tokens up to the current time, while full tokens attend to partial-history tokens up to their time index and to themselves, but never to other full tokens, preventing duplicated static information from propagating along time.
• Figure 5: Action-aware relative attention bias (RAB) with efficient computation. Left: a causal mask with dual sliding windows, which limits each query to attend only to recent past tokens visible within the sliding-window. Right: the action-aware relative encoding pipeline: relative time, position, and action signals are bucketized (as needed), embedded, summed, and injected to the attention logits.