Zenith: Scaling up Ranking Models for Billion-scale Livestreaming Recommendation

TL;DR

Zenith addresses the challenge of scaling ranking models for billion-scale livestreaming by introducing Prime Tokens and two specialized modules, Token Fusion and Token Boost, to efficiently model feature interactions. The approach employs RSA or TMHSA for cross-token interactions and TSwiGLU or TSMoE for tokenwise refinement, preserving token heterogeneity as models grow. Across offline and online evaluations on TikTok Live, Zenith consistently surpasses strong baselines in AUC and logloss and delivers meaningful improvements in engagement metrics, demonstrating practical scalability and deployment viability. The work highlights token heterogeneity as a key factor in scaling up ranking models and provides detailed optimizations for training and inference that enable real-time performance in large-scale systems.

Abstract

Accurately capturing feature interactions is essential in recommender systems, and recent trends show that scaling up model capacity could be a key driver for next-level predictive performance. While prior work has explored various model architectures to capture multi-granularity feature interactions, relatively little attention has been paid to efficient feature handling and scaling model capacity without incurring excessive inference latency. In this paper, we address this by presenting Zenith, a scalable and efficient ranking architecture that learns complex feature interactions with minimal runtime overhead. Zenith is designed to handle a few high-dimensional Prime Tokens with Token Fusion and Token Boost modules, which exhibits superior scaling laws compared to other state-of-the-art ranking methods, thanks to its improved token heterogeneity. Its real-world effectiveness is demonstrated by deploying the architecture to TikTok Live, a leading online livestreaming platform that attracts billions of users globally. Our A/B test shows that Zenith achieves +1.05%/-1.10% in online CTR AUC and Logloss, and realizes +9.93% gains in Quality Watch Session / User and +8.11% in Quality Watch Duration / User.

Figures (4)

• Figure 1: High-level structure of the proposed Zenith architecture. Zenith employs a stack of layers, and each layer has a Token Fusion module and a Token Boost module. Detailed instantiations of these modules are shown in \ref{['fig:zenith_layers']}.
• Figure 2: Designs of Token Fusion and Token Boost modules for Zenith and Zenith++.
• Figure 3: Model performance comparison between our proposed architectures and the baselines. Note that in (a), for Zenith++, we also report the effective parameter count per inference—denoted Zenith++(act)—which reflects the reduced active parameters enabled by our Tokenwise Sparse MoE. All LogLoss values are shown against the baseline DCN-V2 model.
• Figure 4: The comparison of token similarity with and without tokenwise operations in the Token Boost network for Zenith (TSwiGLU) and Zenith++ (TSMoE). The similarity level is defined by the absolute cosine similarity of the token pair, and the deeper red color indicates more similar output, which is detrimental for the deeper layer's expressiveness, limiting model scaling capability. We sampled the first 10 tokens for clarity, and used the average token similarity of the output tokens from the third layer in each model.