Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Efficient Cross-Architecture Knowledge Transfer for Large-Scale Online User Response Prediction

Yucheng Wu, Yuekui Yang, Hongzheng Li, Anan Liu, Jian Xiao, Junjie Zhai, Huan Yu, Shaoping Ma, Leye Wang

TL;DR

CrossAdapt tackles the high switching cost of deploying new architectures in large-scale online user response prediction by separating offline embedding inheritance from online adaptation. It introduces a dimension-adaptive embedding projection with theoretical guarantees, a progressive distillation of interaction networks, and strategic sampling to transfer knowledge efficiently across heterogeneous architectures. The online phase uses asymmetric teacher–student co-distillation with distribution-shift awareness to balance stability and rapid adaptation, yielding consistent AUC gains and substantial training-time reductions on public benchmarks, plus successful industrial deployment at Tencent WeChat Channels. The framework significantly mitigates degradation in AUC, LogLoss, and pCVR bias during model updates, enabling faster, safer adoption of advanced architectures in production systems.

Abstract

Deploying new architectures in large-scale user response prediction systems incurs high model switching costs due to expensive retraining on massive historical data and performance degradation under data retention constraints. Existing knowledge distillation methods struggle with architectural heterogeneity and the prohibitive cost of transferring large embedding tables. We propose CrossAdapt, a two-stage framework for efficient cross-architecture knowledge transfer. The offline stage enables rapid embedding transfer via dimension-adaptive projections without iterative training, combined with progressive network distillation and strategic sampling to reduce computational cost. The online stage introduces asymmetric co-distillation, where students update frequently while teachers update infrequently, together with a distribution-aware adaptation mechanism that dynamically balances historical knowledge preservation and fast adaptation to evolving data. Experiments on three public datasets show that CrossAdapt achieves 0.27-0.43% AUC improvements while reducing training time by 43-71%. Large-scale deployment on Tencent WeChat Channels (~10M daily samples) further demonstrates its effectiveness, significantly mitigating AUC degradation, LogLoss increase, and prediction bias compared to standard distillation baselines.

Efficient Cross-Architecture Knowledge Transfer for Large-Scale Online User Response Prediction

TL;DR

CrossAdapt tackles the high switching cost of deploying new architectures in large-scale online user response prediction by separating offline embedding inheritance from online adaptation. It introduces a dimension-adaptive embedding projection with theoretical guarantees, a progressive distillation of interaction networks, and strategic sampling to transfer knowledge efficiently across heterogeneous architectures. The online phase uses asymmetric teacher–student co-distillation with distribution-shift awareness to balance stability and rapid adaptation, yielding consistent AUC gains and substantial training-time reductions on public benchmarks, plus successful industrial deployment at Tencent WeChat Channels. The framework significantly mitigates degradation in AUC, LogLoss, and pCVR bias during model updates, enabling faster, safer adoption of advanced architectures in production systems.

Abstract

Deploying new architectures in large-scale user response prediction systems incurs high model switching costs due to expensive retraining on massive historical data and performance degradation under data retention constraints. Existing knowledge distillation methods struggle with architectural heterogeneity and the prohibitive cost of transferring large embedding tables. We propose CrossAdapt, a two-stage framework for efficient cross-architecture knowledge transfer. The offline stage enables rapid embedding transfer via dimension-adaptive projections without iterative training, combined with progressive network distillation and strategic sampling to reduce computational cost. The online stage introduces asymmetric co-distillation, where students update frequently while teachers update infrequently, together with a distribution-aware adaptation mechanism that dynamically balances historical knowledge preservation and fast adaptation to evolving data. Experiments on three public datasets show that CrossAdapt achieves 0.27-0.43% AUC improvements while reducing training time by 43-71%. Large-scale deployment on Tencent WeChat Channels (~10M daily samples) further demonstrates its effectiveness, significantly mitigating AUC degradation, LogLoss increase, and prediction bias compared to standard distillation baselines.
Paper Structure (42 sections, 1 theorem, 22 equations, 6 figures, 8 tables, 1 algorithm)

This paper contains 42 sections, 1 theorem, 22 equations, 6 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Problem Formulation
  3. CrossAdapt Framework
  4. Offline Cross-Architecture Transfer
  5. Dimension-Adaptive Embedding Table Projection
  6. Progressive Interaction Network Distillation
  7. Strategic Distillation Sample Selection
  8. Online Adaptive Co-Distillation
  9. Asymmetric Teacher-Student Co-evolution
  10. Adaptive Historical Knowledge Preservation
  11. Offline Evaluation
  12. Experimental Setup
  13. Training Protocol
  14. Baseline Methods
  15. Evaluation Metrics
  16. ...and 27 more sections

Key Result

Theorem 1

Let $\bar{E}_T = E_T - \mathbf{1}_V \mu^\top$ be the centered teacher embedding matrix and $C = \frac{1}{V}\bar{E}_T^\top \bar{E}_T$ be its covariance matrix with eigendecomposition $C = U\Lambda U^\top$ where $\Lambda = \mathrm{diag}(\lambda_1, \ldots, \lambda_{d_T})$ and $\lambda_1 \geq \cdots \ge

Figures (6)

  • Figure 1: A typical user response prediction model architecture, composed of embedding tables and feature interaction networks.
  • Figure 2: Overview of CrossAdapt with offline cross-architecture transfer and online adaptive co-distillation.
  • Figure 3: Strategic distillation sample selection. Left: Sample space composition showing clicked samples for standard pCVR training and the augmentation with unclicked samples. Right: Sampling pipeline applying temporal diversity sampling followed by class-balanced sampling to construct the training set.
  • Figure 4: Convergence curves on Avazu. CrossAdapt-Sample achieves faster convergence compared to baselines.
  • Figure 5: Online training convergence curves on Avazu and Criteo1T. CrossAdapt achieves faster convergence compared to baselines.
  • ...and 1 more figures

Theorems & Definitions (4)

  • Definition 1: Model Architecture
  • Definition 2: Model Switching Cost
  • Theorem 1: Optimal Embedding Projections
  • proof