Scaling Sequential Recommendation Models with Transformers

TL;DR

The paper studies how transformer-based sequential recommendation models scale with data and parameter count, testing whether scaling laws similar to those in NLP apply. It introduces the Scalable Recommendation Transformer (SRT), which replaces item ID embeddings with a fixed feature extractor, allowing the model size to grow independently of catalog size and enabling pre-training followed by task-specific fine-tuning. The authors derive empirical scaling laws, propose compute-based and data-based performance estimators, and demonstrate these ideas on the Amazon Product Data at scale, including transfer learning across domains. The work provides practical guidance for designing and deploying large, high-dimensional sequential recommender systems, and shows that pre-trained transformers can yield substantial gains over training-from-scratch baselines.

Abstract

Modeling user preferences has been mainly addressed by looking at users' interaction history with the different elements available in the system. Tailoring content to individual preferences based on historical data is the main goal of sequential recommendation. The nature of the problem, as well as the good performance observed across various domains, has motivated the use of the transformer architecture, which has proven effective in leveraging increasingly larger amounts of training data when accompanied by an increase in the number of model parameters. This scaling behavior has brought a great deal of attention, as it provides valuable guidance in the design and training of even larger models. Taking inspiration from the scaling laws observed in training large language models, we explore similar principles for sequential recommendation. We use the full Amazon Product Data dataset, which has only been partially explored in other studies, and reveal scaling behaviors similar to those found in language models. Compute-optimal training is possible but requires a careful analysis of the compute-performance trade-offs specific to the application. We also show that performance scaling translates to downstream tasks by fine-tuning larger pre-trained models on smaller task-specific domains. Our approach and findings provide a strategic roadmap for model training and deployment in real high-dimensional preference spaces, facilitating better training and inference efficiency. We hope this paper bridges the gap between the potential of transformers and the intrinsic complexities of high-dimensional sequential recommendation in real-world recommender systems. Code and models can be found at https://github.com/mercadolibre/srt

Figures (7)

• Figure 1: Increase in the catalog size induced by an increase in the number of active users (top) and interactions (middle). Increasing the number of items increases the total parameter count (bottom) in models that use trainable item embeddings for 64-dimensional embeddings. For smaller models (33K parameters), the catalog size dominates the total parameter count, while for larger models (19B parameters), the total number of trainable parameters remains stable across a wide spectrum of catalog sizes.
• Figure 2: Traditional transformer-based recommendation models (left) learn vector embeddings for all items in the catalog and access them via table lookups (LUT). The system might also include a classification layer aimed at predicting the index of the target item. On the contrary, we propose to use a fixed (and task-agnostic) feature extractor to encode the items in the catalog (right) and to predict item-to-item similarities using the output embeddings of the transformer model. We use the last element of the sequence as target and try to match its id (classification) or input embedding (regression).
• Figure 3: Distribution of the number of reviews per item in the Amazon beauty dataset for the full and 5-core versions. Similar behaviors are observed in Amazon sports.
• Figure 4: NDCG@5 vs FLOPs for different runs with different training set sizes and model complexities. The colormap of each plot encodes the number of training interactions (left) and the number of non-embedding parameters (right).
• Figure 5: Number of seen iterations (left) and number of non-embedding parameters as a function of the FLOP count for the points of maximal performance. Color encodes the NDCG@5 score of each configuration. The dotted red lines show linear fit curves of the corresponding point cloud in log-log space.