Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Query-Mixed Interest Extraction and Heterogeneous Interaction: A Scalable CTR Model for Industrial Recommender Systems

Fangye Wang, Guowei Yang, Xiaojiang Zhou, Song Yang, Pengjie Wang

TL;DR

HeMix is proposed, a scalable ranking model that unifies adaptive sequence tokenization and heterogeneous interaction structure that scales effectively and consistently outperforms strong baselines on industrial-scale datasets and has been deployed on the AMAP platform.

Abstract

Learning effective feature interactions is central to modern recommender systems, yet remains challenging in industrial settings due to sparse multi-field inputs and ultra-long user behavior sequences. While recent scaling efforts have improved model capacity, they often fail to construct both context-aware and context-independent user intent from the long-term and real-time behavior sequence. Meanwhile, recent work also suffers from inefficient and homogeneous interaction mechanisms, leading to suboptimal prediction performance. To address these limitations, we propose HeMix, a scalable ranking model that unifies adaptive sequence tokenization and heterogeneous interaction structure. Specifically, HeMix introduces a Query-Mixed Interest Extraction module that jointly models context-aware and context-independent user interests via dynamic and fixed queries over global and real-time behavior sequences. For interaction, we replace self-attention with the HeteroMixer block, enabling efficient, multi-granularity cross-feature interactions that adopt the multi-head token fusion, heterogeneous interaction and group-aligned reconstruction pipelines. HeMix demonstrates favorable scaling behavior, driven by the HeteroMixer block, where increasing model scale via parameter expansion leads to steady improvements in recommendation accuracy. Experiments on industrial-scale datasets show that HeMix scales effectively and consistently outperforms strong baselines. Most importantly, HeMix has been deployed on the AMAP platform, delivering significant online gains: +0.61% GMV, +2.32% PV_CTR, and +0.81% UV_CVR.

Query-Mixed Interest Extraction and Heterogeneous Interaction: A Scalable CTR Model for Industrial Recommender Systems

TL;DR

HeMix is proposed, a scalable ranking model that unifies adaptive sequence tokenization and heterogeneous interaction structure that scales effectively and consistently outperforms strong baselines on industrial-scale datasets and has been deployed on the AMAP platform.

Abstract

Learning effective feature interactions is central to modern recommender systems, yet remains challenging in industrial settings due to sparse multi-field inputs and ultra-long user behavior sequences. While recent scaling efforts have improved model capacity, they often fail to construct both context-aware and context-independent user intent from the long-term and real-time behavior sequence. Meanwhile, recent work also suffers from inefficient and homogeneous interaction mechanisms, leading to suboptimal prediction performance. To address these limitations, we propose HeMix, a scalable ranking model that unifies adaptive sequence tokenization and heterogeneous interaction structure. Specifically, HeMix introduces a Query-Mixed Interest Extraction module that jointly models context-aware and context-independent user interests via dynamic and fixed queries over global and real-time behavior sequences. For interaction, we replace self-attention with the HeteroMixer block, enabling efficient, multi-granularity cross-feature interactions that adopt the multi-head token fusion, heterogeneous interaction and group-aligned reconstruction pipelines. HeMix demonstrates favorable scaling behavior, driven by the HeteroMixer block, where increasing model scale via parameter expansion leads to steady improvements in recommendation accuracy. Experiments on industrial-scale datasets show that HeMix scales effectively and consistently outperforms strong baselines. Most importantly, HeMix has been deployed on the AMAP platform, delivering significant online gains: +0.61% GMV, +2.32% PV_CTR, and +0.81% UV_CVR.
Paper Structure (24 sections, 21 equations, 3 figures, 5 tables)

This paper contains 24 sections, 21 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methodology
  4. Overall Architecture
  5. Input Layer and Feature Embedding
  6. Feature Tokenization
  7. Non-sequential features Tokenization.
  8. Query-mixed Interest Extraction for Sequence Tokenization.
  9. Interaction Layer with HeteroMixer Block
  10. HeteroMixing layer
  11. Heterogeneous Feed-Forward Network
  12. Prediction Layer
  13. Scaling Up the HeMix
  14. Experiments
  15. Experiment Settings
  16. ...and 9 more sections

Figures (3)

  • Figure 1: The overall architecture of HeMix with the feature embedding& tokenization layer and the HeteroMixing module.
  • Figure 2: The structure of Mixed Hetero Attention, which applies the heterogeneous projection matrix for each query vector, and shares the projection for key and value(i.e., Seq. Features), respectively.
  • Figure 3: Scaling performance between CTR_CTR and CVR_AUC and the #Param/GFlops of two SOTA models. The x-axis adopts a logarithmic scale.