Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Feature Staleness Aware Incremental Learning for CTR Prediction

Zhikai Wang, Yanyan Shen, Zibin Zhang, Kangyi Lin

TL;DR

This work tackles feature staleness in incremental CTR prediction by introducing FeSAIL, a two-component framework consisting of SAS and SAR. SAS uses a fixed-size reservoir and a greedy, $1-\frac{1}{e}$-approximate algorithm to selectively replay stale-feature samples, while SAR imposes a per-feature regularization that scales with feature staleness to restrict embedding updates. Across four public and private datasets, FeSAIL consistently outperforms state-of-the-art incremental methods, achieving about $1.21\%$ average AUC improvement and offering competitive training efficiency through neighbor-based optimization. The approach is model-agnostic and practical for real-world online systems, providing a scalable solution to mitigate feature staleness effects in CTR models.

Abstract

Click-through Rate (CTR) prediction in real-world recommender systems often deals with billions of user interactions every day. To improve the training efficiency, it is common to update the CTR prediction model incrementally using the new incremental data and a subset of historical data. However, the feature embeddings of a CTR prediction model often get stale when the corresponding features do not appear in current incremental data. In the next period, the model would have a performance degradation on samples containing stale features, which we call the feature staleness problem. To mitigate this problem, we propose a Feature Staleness Aware Incremental Learning method for CTR prediction (FeSAIL) which adaptively replays samples containing stale features. We first introduce a staleness aware sampling algorithm (SAS) to sample a fixed number of stale samples with high sampling efficiency. We then introduce a staleness aware regularization mechanism (SAR) for a fine-grained control of the feature embedding updating. We instantiate FeSAIL with a general deep learning-based CTR prediction model and the experimental results demonstrate FeSAIL outperforms various state-of-the-art methods on four benchmark datasets.

Feature Staleness Aware Incremental Learning for CTR Prediction

TL;DR

This work tackles feature staleness in incremental CTR prediction by introducing FeSAIL, a two-component framework consisting of SAS and SAR. SAS uses a fixed-size reservoir and a greedy, -approximate algorithm to selectively replay stale-feature samples, while SAR imposes a per-feature regularization that scales with feature staleness to restrict embedding updates. Across four public and private datasets, FeSAIL consistently outperforms state-of-the-art incremental methods, achieving about average AUC improvement and offering competitive training efficiency through neighbor-based optimization. The approach is model-agnostic and practical for real-world online systems, providing a scalable solution to mitigate feature staleness effects in CTR models.

Abstract

Click-through Rate (CTR) prediction in real-world recommender systems often deals with billions of user interactions every day. To improve the training efficiency, it is common to update the CTR prediction model incrementally using the new incremental data and a subset of historical data. However, the feature embeddings of a CTR prediction model often get stale when the corresponding features do not appear in current incremental data. In the next period, the model would have a performance degradation on samples containing stale features, which we call the feature staleness problem. To mitigate this problem, we propose a Feature Staleness Aware Incremental Learning method for CTR prediction (FeSAIL) which adaptively replays samples containing stale features. We first introduce a staleness aware sampling algorithm (SAS) to sample a fixed number of stale samples with high sampling efficiency. We then introduce a staleness aware regularization mechanism (SAR) for a fine-grained control of the feature embedding updating. We instantiate FeSAIL with a general deep learning-based CTR prediction model and the experimental results demonstrate FeSAIL outperforms various state-of-the-art methods on four benchmark datasets.
Paper Structure (15 sections, 3 theorems, 4 equations, 6 figures, 3 tables, 1 algorithm)

This paper contains 15 sections, 3 theorems, 4 equations, 6 figures, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Preliminaries
  3. The FeSAIL Approach
  4. Overview
  5. Staleness Aware Sampling (SAS)
  6. Staleness Aware Regularization (SAR)
  7. Implementation Details
  8. Complexity Analysis
  9. Experiment
  10. Experimental Setups
  11. Experimental Results: RQ1
  12. Ablation Study: RQ2
  13. Parameter Sensitivity: RQ3
  14. Case Study: RQ4
  15. Conclusion

Key Result

Lemma 1

In $l+1^{th}$ iteration, we always have $W_{l+1}\ge\frac{c_l}{L}$, where $W_{l+1}$ is a possible total weight.

Figures (6)

  • Figure 1: The observed feature staleness problem on Avazu.
  • Figure 2: The overview of FeSAIL. All features that appear in the current dataset are in green squares. The redness of a feature represents the extent of its staleness.
  • Figure 3: Prediction performance on each time span. We present the six most competitive baselines and omit the remaining methods.
  • Figure 4: Ablation study on Avazu.
  • Figure 5: Parameter sensitivity w.r.t different inverse correlation functions and biases on Avazu.
  • ...and 1 more figures

Theorems & Definitions (8)

  • Example 1
  • Definition 1: Stale Features Sampling (SFS)
  • Lemma 1
  • proof
  • Lemma 2
  • proof
  • Theorem 1
  • proof