Practice on Long Behavior Sequence Modeling in Tencent Advertising

TL;DR

This work tackles long-sequence modeling in a sparse, cross-domain advertising setting by building unified commercial behavior trajectories that fuse ad and content-domain signals. It introduces a two-stage framework: a search stage with hierarchical hard search and DARE decoupled embeddings, and a sequence modeling stage featuring Decoupled Side Info TIN (DSI-TIN), Target-Decoupled Temporal Encoding (TdPE), Target-Decoupled SASRec, and Stacked TIN (STIN) to capture high-order interactions. SPU construction unifies product representations across diverse items, while platform innovations (prefetching, multi-streaming, CUDA Graphs, mixed precision, and embedding caching) enable production-scale deployment. In production across WeChat Channels and Moments, these methods deliver significant GMV lifts (e.g., 4.22% and 1.96%, respectively), validating the practicality and impact of cross-domain long-sequence modeling in large-scale advertising. The work demonstrates a cohesive pipeline from data consolidation to modeling and platform optimization, establishing a scalable blueprint for long-horizon user-interest modeling in industry.

Abstract

Long-sequence modeling has become an indispensable frontier in recommendation systems for capturing users' long-term preferences. However, user behaviors within advertising domains are inherently sparse, posing a significant barrier to constructing long behavioral sequences using data from a single advertising domain alone. This motivates us to collect users' behaviors not only across diverse advertising scenarios, but also beyond the boundaries of the advertising domain into content domains-thereby constructing unified commercial behavior trajectories. This cross-domain or cross-scenario integration gives rise to the following challenges: (1) feature taxonomy gaps between distinct scenarios and domains, (2) inter-field interference arising from irrelevant feature field pairs, and (3) target-wise interference in temporal and semantic patterns when optimizing for different advertising targets. To address these challenges, we propose several practical approaches within the two-stage framework for long-sequence modeling. In the first (search) stage, we design a hierarchical hard search method for handling complex feature taxonomy hierarchies, alongside a decoupled embedding-based soft search to alleviate conflicts between attention mechanisms and feature representation. In the second (sequence modeling) stage, we introduce: (a) Decoupled Side Information Temporal Interest Networks (TIN) to mitigate inter-field conflicts; (b) Target-Decoupled Positional Encoding and Target-Decoupled SASRec to address target-wise interference; and (c) Stacked TIN to model high-order behavioral correlations. Deployed in production on Tencent's large-scale advertising platforms, our innovations delivered significant performance gains: an overall 4.22% GMV lift in WeChat Channels and an overall 1.96% GMV increase in WeChat Moments.

Figures (8)

• Figure 1: Illustration of building up a long sequence for advertisement recommendation.
• Figure 2: Illustration of the TIN mechanism regarding one behavior. It consists of the target-aware attention, target-aware representation, and target-aware temporal encoding.
• Figure 3: Learned temporal patterns of our production model. The x-axis denotes each target-aware position (left) or time interval in days between each behavior and the target (right), the y-axis denotes the L1 norm of the corresponding temporal embedding. There is stronger decaying pattern in the time interval embeddings.
• Figure 4: Architecture of Decoupled Side Info TIN. We build multiple TINs, each with a different set of features in the attention module. All TINs use all the side features in their representation.
• Figure 5: Visualization of the outputs from the vanilla SASRec user tower (left) and the Target decoupled SASRec tower (right) through t-SNE. Colors denote ad categories. There is a better clustering pattern in our Target-aware SASRec.