Deep Pattern Network for Click-Through Rate Prediction

TL;DR

The paper tackles CTR prediction by arguing that user behavior patterns contain rich, high-order information that standard co-occurrence and sequential models miss. It introduces Deep Pattern Network (DPN), which combines target-aware pattern retrieval (TPRM), self-supervised pattern refinement (SPRM), and target pattern attention (TPA) to model pattern-level dependencies. Across three real datasets, DPN consistently outperforms strong baselines and remains compatible with multiple backbones, with ablations confirming the critical roles of retrieval, refinement, and attention. The work demonstrates that refining and interrelating user behavior patterns can substantially boost CTR and provides a framework for pattern-centric interest modeling in recommender systems.

Abstract

Click-through rate (CTR) prediction tasks play a pivotal role in real-world applications, particularly in recommendation systems and online advertising. A significant research branch in this domain focuses on user behavior modeling. Current research predominantly centers on modeling co-occurrence relationships between the target item and items previously interacted with by users in their historical data. However, this focus neglects the intricate modeling of user behavior patterns. In reality, the abundance of user interaction records encompasses diverse behavior patterns, indicative of a spectrum of habitual paradigms. These patterns harbor substantial potential to significantly enhance CTR prediction performance. To harness the informational potential within user behavior patterns, we extend Target Attention (TA) to Target Pattern Attention (TPA) to model pattern-level dependencies. Furthermore, three critical challenges demand attention: the inclusion of unrelated items within behavior patterns, data sparsity in behavior patterns, and computational complexity arising from numerous patterns. To address these challenges, we introduce the Deep Pattern Network (DPN), designed to comprehensively leverage information from user behavior patterns. DPN efficiently retrieves target-related user behavior patterns using a target-aware attention mechanism. Additionally, it contributes to refining user behavior patterns through a pre-training paradigm based on self-supervised learning while promoting dependency learning within sparse patterns. Our comprehensive experiments, conducted across three public datasets, substantiate the superior performance and broad compatibility of DPN.

Figures (10)

• Figure 1: Illustration of the dependency modeling that the existing SOTA methods and DPN focus on. The red dotted line indicates Target Attention or Target Pattern Attention.
• Figure 2: Statistical results for 3-order user behavior patterns in the Taobao dataset (log-scale plot). Each scatter point with coordinates $(x,y)$ represents that the number of 3-order user behavior patterns with occurrence frequency $x$ in the Taobao dataset is $y$. The figure shows the Top-5 high-frequency user behavior patterns for items and categories, respectively.
• Figure 3: Overall architecture of the proposed DPN. Target-ware Pattern Retrieval Model (TPRM) searches the Top-K target-related pattern from user behavior sequence according to target attention scores. SPRM adopts a pre-trained refinement network to perform fine-grained denoising of the behavior patterns, whose detailed illustration is shown in Figure \ref{['fig:SPRM']}. TPA models the dependency between the target behavior pattern and refined historical behavior patterns.
• Figure 4: Illustration of Self-supervised Pattern Refinement Module (SPRM). On the left, random data augmentation and denoising objectives are applied to pretrain refinement network based on self-supervised learning. On the right, the refinement network is utilized to generate a logit vector for pattern refinement in CTR prediction tasks.
• Figure 5: Performances with different numbers of retrieved patterns $K$.

Theorems & Definitions (1)

• definition 1