LLaTTE: Scaling Laws for Multi-Stage Sequence Modeling in Large-Scale Ads Recommendation

TL;DR

LLaTTE demonstrates that multi-stage sequence modeling in large-scale ads recommendation obeys power-law scaling with compute, depth, and sequence length, and that semantic content features are essential to unlock these gains. By splitting computation into a heavy upstream encoder and a lightweight online ranker, the approach achieves substantial production gains (e.g., 4.3% conversion uplift) while maintaining low latency. The study systematically maps how model capacity, horizon, and data richness interact, introduces the Transfer Ratio to quantify cross-stage benefits, and shows that content-rich signals are necessary to bend scaling curves in favor of deeper architectures. The resulting framework provides a practical blueprint for applying scaling laws to industrial recommender systems, with clear guidance for compute allocation and deployment strategies.

Abstract

We present LLaTTE (LLM-Style Latent Transformers for Temporal Events), a scalable transformer architecture for production ads recommendation. Through systematic experiments, we demonstrate that sequence modeling in recommendation systems follows predictable power-law scaling similar to LLMs. Crucially, we find that semantic features bend the scaling curve: they are a prerequisite for scaling, enabling the model to effectively utilize the capacity of deeper and longer architectures. To realize the benefits of continued scaling under strict latency constraints, we introduce a two-stage architecture that offloads the heavy computation of large, long-context models to an asynchronous upstream user model. We demonstrate that upstream improvements transfer predictably to downstream ranking tasks. Deployed as the largest user model at Meta, this multi-stage framework drives a 4.3\% conversion uplift on Facebook Feed and Reels with minimal serving overhead, establishing a practical blueprint for harnessing scaling laws in industrial recommender systems.

Figures (6)

• Figure 1: LLaTTE architecture overview. We hold the non-sequence backbone (DHEN) and task heads fixed, and scale the transformer-based sequence module in depth, width, and sequence length. The sequence module utilizes Multi-head Latent Attention (MLA) and adaptive pyramidal trimming to efficiently process long user histories.
• Figure 2: Sequence length scaling for $L=1$, $L=2$, and $L=4$ models. NE improves smoothly with longer histories, with deeper models exhibiting a steeper dependence on $T$.
• Figure 3: Attention-score distribution across the sequence. The model continues to attend to events throughout the history, supporting the usefulness of long contexts.
• Figure 4: Relative normalized entropy (NE) gain versus training compute (FLOPs, log scale) for different scaling strategies. The slope of each line indicates NE improvement per $10\times$ increase in compute.
• Figure 5: Total attention weight distributed to event tokens bucketized by hours prior to the user request.