Massive Memorization with Hundreds of Trillions of Parameters for Sequential Transducer Generative Recommenders

TL;DR

The paper tackles the scalability bottlenecks in industrial-scale recommender systems when handling ultra-long user histories by introducing VISTA, a two-stage framework that first compresses long histories into a compact set of embeddings and then performs target-aware attention using these cached summaries. A quasi-linear attention mechanism (QLA) enhances the memory and computation efficiency of the summarization stage, while a generative reconstruction loss improves memorization of user histories. An embedding delivery system enables scalable, real-time deployment by caching summarization embeddings in a large key-value store and streaming updates via distributed pipelines. Empirically, VISTA achieves strong offline improvements on public and industrial datasets and delivers measurable online gains in watch time and engagement, validating its practical impact for production-scale recommender systems.

Abstract

Modern large-scale recommendation systems rely heavily on user interaction history sequences to enhance the model performance. The advent of large language models and sequential modeling techniques, particularly transformer-like architectures, has led to significant advancements recently (e.g., HSTU, SIM, and TWIN models). While scaling to ultra-long user histories (10k to 100k items) generally improves model performance, it also creates significant challenges on latency, queries per second (QPS) and GPU cost in industry-scale recommendation systems. Existing models do not adequately address these industrial scalability issues. In this paper, we propose a novel two-stage modeling framework, namely VIrtual Sequential Target Attention (VISTA), which decomposes traditional target attention from a candidate item to user history items into two distinct stages: (1) user history summarization into a few hundred tokens; followed by (2) candidate item attention to those tokens. These summarization token embeddings are then cached in storage system and then utilized as sequence features for downstream model training and inference. This novel design for scalability enables VISTA to scale to lifelong user histories (up to one million items) while keeping downstream training and inference costs fixed, which is essential in industry. Our approach achieves significant improvements in offline and online metrics and has been successfully deployed on an industry leading recommendation platform serving billions of users.

Figures (14)

• Figure 1: VISTA replaces standard attention with a two-stage process, allowing downstream models to compute only the highly efficient second stage.
• Figure 2: An overview of VISTA architecture.
• Figure 3: Visualization of UIH summarization embeddings.
• Figure 4: The QLU module.
• Figure 5: Generative reconstruction loss.