SMES: Towards Scalable Multi-Task Recommendation via Expert Sparsity

TL;DR

SMES addresses the mismatch between uniform model scaling and heterogeneous task capacity in industrial multi-task recommendations by introducing progressive expert routing that partitions activation into a shared expert subset and task-adaptive experts, keeping per-instance active experts bounded. A global multi-task load-balancing regularizer mitigates cross-task expert hotspots, enabling scalable and stable training as the expert pool grows. Deployed in a large-scale setting, SMES yields consistent online gains (e.g., GAUC improvements and watch-time uplift) while maintaining strict latency budgets, outperforming dense MoE and naive sparse baselines. The approach combines algorithmic innovations with deployment optimizations (deduplicated execution, reindexed grouped GEMM, profiling-guided memory management) to achieve practical, low-latency scaling for industrial recommender systems.

Abstract

Industrial recommender systems typically rely on multi-task learning to estimate diverse user feedback signals and aggregate them for ranking. Recent advances in model scaling have shown promising gains in recommendation. However, naively increasing model capacity imposes prohibitive online inference costs and often yields diminishing returns for sparse tasks with skewed label distributions. This mismatch between uniform parameter scaling and heterogeneous task capacity demands poses a fundamental challenge for scalable multi-task recommendation. In this work, we investigate parameter sparsification as a principled scaling paradigm and identify two critical obstacles when applying sparse Mixture-of-Experts (MoE) to multi-task recommendation: exploded expert activation that undermines instance-level sparsity and expert load skew caused by independent task-wise routing. To address these challenges, we propose SMES, a scalable sparse MoE framework with progressive expert routing. SMES decomposes expert activation into a task-shared expert subset jointly selected across tasks and task-adaptive private experts, explicitly bounding per-instance expert execution while preserving task-specific capacity. In addition, SMES introduces a global multi-gate load-balancing regularizer that stabilizes training by regulating aggregated expert utilization across all tasks. SMES has been deployed in Kuaishou large-scale short-video services, supporting over 400 million daily active users. Extensive online experiments demonstrate stable improvements, with GAUC gain of 0.29% and a 0.31% uplift in user watch time.

Figures (5)

• Figure 1: Performance trends across tasks in KuaiRand dataset. The curves show AUC (solid line) and GAUC (dashed line) trends with increasing model parameter scale (billion). For tasks (e.g., watching-time task,Effective-view), performance consistently improves with increasing parameter scale. In contrast, for tasks (e.g., interaction tasks such as like and follow), blindly increasing model capacity can lead to performance degradation.
• Figure 2: The architecture of our SMES for multi-task recommendation. It contains two key components: (1) Progressive Expert Routing uses a task-shared router and task-adaptive sub-routers to select experts (only Effective-view and Long-view tasks are visualized for clarity). (2) Deduplicated Expert Execution removes redundant expert computations across tasks.
• Figure 3: Scalability Analysis of SMES vs. Dense MoE: Performance and Latency in Effective View and Click Tasks.
• Figure 4: Hyper-Parameter Sensitivity of SMES. We evaluate the impact of hidden dimensions and expert numbers on SMES performance across core recommendation tasks (Effective-view, Like, Follow, Comment).
• Figure 5: Task-Expert Activation of our SMES. For the sake of clarity and conciseness, we only display three most-activated with a high number of assigned experts, marked in orange, and three least-activated tasks with a low number, marked in blue.