Actions Speak Louder than Words: Trillion-Parameter Sequential Transducers for Generative Recommendations

TL;DR

The paper tackles the scalability of industrial recommendation systems operating on billions of heterogeneous features and user actions. It introduces Generative Recommenders (GRs) and a high-performance HSTU encoder to treat user actions as a generative, sequential transduction problem, unifying feature spaces and enabling target-aware inference in streaming settings. Key contributions include the HSTU architecture with pointwise aggregated attention, sparsity-driven efficiency techniques, memory-optimized design, and M-FALCON for cost-aware, batched inference, culminating in up to 12.4% online gains and substantial throughput improvements over DLRMs. A core finding is that GRs exhibit a power-law scaling with training compute across three orders of magnitude, suggesting a path toward foundation-model-scale recommendations with reduced carbon footprint and broader applicability across recommendation, search, and ads. The work is validated on synthetic, public, and real industrial deployments, demonstrating both theoretical and practical advantages of treating recommendations as generative, sequence-based tasks with a unified feature space.

Abstract

Large-scale recommendation systems are characterized by their reliance on high cardinality, heterogeneous features and the need to handle tens of billions of user actions on a daily basis. Despite being trained on huge volume of data with thousands of features, most Deep Learning Recommendation Models (DLRMs) in industry fail to scale with compute. Inspired by success achieved by Transformers in language and vision domains, we revisit fundamental design choices in recommendation systems. We reformulate recommendation problems as sequential transduction tasks within a generative modeling framework ("Generative Recommenders"), and propose a new architecture, HSTU, designed for high cardinality, non-stationary streaming recommendation data. HSTU outperforms baselines over synthetic and public datasets by up to 65.8% in NDCG, and is 5.3x to 15.2x faster than FlashAttention2-based Transformers on 8192 length sequences. HSTU-based Generative Recommenders, with 1.5 trillion parameters, improve metrics in online A/B tests by 12.4% and have been deployed on multiple surfaces of a large internet platform with billions of users. More importantly, the model quality of Generative Recommenders empirically scales as a power-law of training compute across three orders of magnitude, up to GPT-3/LLaMa-2 scale, which reduces carbon footprint needed for future model developments, and further paves the way for the first foundational models in recommendations.

Figures (16)

• Figure 1: Total compute used to train deep learning models over the years. DLRM results are from dlrm_isca22; GRs are deployed models from this work. DLRMs/GRs are continuously trained in a streaming setting; we report compute used per year.
• Figure 2: Comparison of features and training procedures: DLRMs vs GRs. $E,F,G,H$ denote categorical features. $\Phi_i$ represents the $i$-th item in the merged main time series. $\Psi_k(t_j)$ denotes training example $k$ emitted at time $t_j$. Full notations can be found in \ref{['sec:app-notations']}.
• Figure 3: Comparison of key model components: DLRMs vs GRs. The complete DLRM setup dlrm_isca22 is shown on the left side and a simplified HSTU is shown on the right.
• Figure 4: Impact of Stochastic Length (SL) on metrics. Left: $n=4096$. Right: $n=8192$. Full results can be found in \ref{['sec:app-stochastic-length']}.
• Figure 5: Training NE.