Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Distribution-Aware End-to-End Embedding for Streaming Numerical Features in Click-Through Rate Prediction

Jiahao Liu, Hongji Ruan, Weimin Zhang, Ziye Tong, Derick Tang, Zhanpeng Zeng, Qinsong Zeng, Peng Zhang, Tun Lu, Ning Gu

TL;DR

DAES tackles the challenge of embedding streaming numerical features for CTR with nonstationary distributions by integrating distributional priors into an end-to-end learning framework. It introduces Jump Reservoir Sampling to efficiently estimate global quantiles, encodes inputs in a quantile space, and applies field-aware modulation to produce context-sensitive meta-embeddings. Empirical results on public benchmarks and industrial data show DAES consistently outperforms state-of-the-art baselines and delivers substantial online gains (e.g., ARPU improvements) while reducing engineering complexity. The approach is scalable, pluggable, and has been deployed in production, demonstrating practical impact for large-scale streaming CTR systems.

Abstract

This paper explores effective numerical feature embedding for Click-Through Rate prediction in streaming environments. Conventional static binning methods rely on offline statistics of numerical distributions; however, this inherently two-stage process often triggers semantic drift during bin boundary updates. While neural embedding methods enable end-to-end learning, they often discard explicit distributional information. Integrating such information end-to-end is challenging because streaming features often violate the i.i.d. assumption, precluding unbiased estimation of the population distribution via the expectation of order statistics. Furthermore, the critical context dependency of numerical distributions is often neglected. To this end, we propose DAES, an end-to-end framework designed to tackle numerical feature embedding in streaming training scenarios by integrating distributional information with an adaptive modulation mechanism. Specifically, we introduce an efficient reservoir-sampling-based distribution estimation method and two field-aware distribution modulation strategies to capture streaming distributions and field-dependent semantics. DAES significantly outperforms existing approaches as demonstrated by extensive offline and online experiments and has been fully deployed on a leading short-video platform with hundreds of millions of daily active users.

Distribution-Aware End-to-End Embedding for Streaming Numerical Features in Click-Through Rate Prediction

TL;DR

DAES tackles the challenge of embedding streaming numerical features for CTR with nonstationary distributions by integrating distributional priors into an end-to-end learning framework. It introduces Jump Reservoir Sampling to efficiently estimate global quantiles, encodes inputs in a quantile space, and applies field-aware modulation to produce context-sensitive meta-embeddings. Empirical results on public benchmarks and industrial data show DAES consistently outperforms state-of-the-art baselines and delivers substantial online gains (e.g., ARPU improvements) while reducing engineering complexity. The approach is scalable, pluggable, and has been deployed in production, demonstrating practical impact for large-scale streaming CTR systems.

Abstract

This paper explores effective numerical feature embedding for Click-Through Rate prediction in streaming environments. Conventional static binning methods rely on offline statistics of numerical distributions; however, this inherently two-stage process often triggers semantic drift during bin boundary updates. While neural embedding methods enable end-to-end learning, they often discard explicit distributional information. Integrating such information end-to-end is challenging because streaming features often violate the i.i.d. assumption, precluding unbiased estimation of the population distribution via the expectation of order statistics. Furthermore, the critical context dependency of numerical distributions is often neglected. To this end, we propose DAES, an end-to-end framework designed to tackle numerical feature embedding in streaming training scenarios by integrating distributional information with an adaptive modulation mechanism. Specifically, we introduce an efficient reservoir-sampling-based distribution estimation method and two field-aware distribution modulation strategies to capture streaming distributions and field-dependent semantics. DAES significantly outperforms existing approaches as demonstrated by extensive offline and online experiments and has been fully deployed on a leading short-video platform with hundreds of millions of daily active users.
Paper Structure (46 sections, 3 theorems, 35 equations, 7 figures, 4 tables, 1 algorithm)

This paper contains 46 sections, 3 theorems, 35 equations, 7 figures, 4 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Click-Through Rate Prediction
  4. Embedding for Numerical Features in CTR Prediction
  5. DAES
  6. Problem Formulation
  7. Distribution Estimation over Data Streams
  8. Reservoir Sampling
  9. Quantile Estimation
  10. Jump Sampling
  11. Quantile Space Interpolation
  12. Sample-to-Quantile Mapping
  13. Quantile Space Interpolation
  14. Efficiency Analysis
  15. Field-Aware Meta-Embedding Aggregation
  16. ...and 31 more sections

Key Result

theorem 1

Let $q_\alpha^{(t)}$ be the true $\alpha$-quantile of $F_t(x)$. As $m \to \infty$, $\hat{q}_\alpha^{(t)}$ converges in probability to $q_\alpha^{(t)}$: $\hat{q}_\alpha^{(t)} \xrightarrow{p} q_\alpha^{(t)}$.

Figures (7)

  • Figure 1: Illustration of the Deep CTR model architecture and the streaming learning system. This paper explores effective numerical feature embedding for CTR prediction in streaming environments.
  • Figure 2: Comparison of different numerical feature embedding paradigms. (a) Static Binning: mapping values to discrete buckets via offline statistics. (b) Neural Embedding: transforming values through differentiable neural layers. (c) Interpolated Binning: aggregating meta-embeddings via linear interpolation with static buckets. (d) Dynamic Quantile Embedding: generating distribution-aware weights using online-estimated quantiles.
  • Figure 3: Performance comparison of interpolation conducted in value space versus quantile space.
  • Figure 4: Impact of the modulation hyperparameter $\beta$ on model performance.
  • Figure 5: Performance analysis of DAES and AutoDis across varying meta-embedding counts and numbers of numerical features on the Criteo dataset.
  • ...and 2 more figures

Theorems & Definitions (4)

  • definition 1: Reservoir-based Quantile Estimator
  • theorem 1: Consistency
  • theorem 2: Distribution of Jump Length
  • lemma 1: Complexity of Jump Sampling