Understanding Scaling Laws for Recommendation Models

TL;DR

The paper investigates how CTR-focused DLRMs scale with data, parameters, and compute by fitting a power-law plus constant model to normalized cross-entropy loss across multiple scaling schemes. It finds that data and compute scaling operate in a high-return regime with modest exponents, while parameter scaling is in a saturated regime with diminishing returns, suggesting data scaling as the primary path for practical gains. The study quantifies scaling efficiency across embedding, overarch, and MLP scaling, revealing no universal winner and highlighting regime-dependent tradeoffs. These insights inform infrastructure planning and point toward data-centric strategies and architecture innovations to sustain performance growth as models scale.

Abstract

Scale has been a major driving force in improving machine learning performance, and understanding scaling laws is essential for strategic planning for a sustainable model quality performance growth, long-term resource planning and developing efficient system infrastructures to support large-scale models. In this paper, we study empirical scaling laws for DLRM style recommendation models, in particular Click-Through Rate (CTR). We observe that model quality scales with power law plus constant in model size, data size and amount of compute used for training. We characterize scaling efficiency along three different resource dimensions, namely data, parameters and compute by comparing the different scaling schemes along these axes. We show that parameter scaling is out of steam for the model architecture under study, and until a higher-performing model architecture emerges, data scaling is the path forward. The key research questions addressed by this study include: Does a recommendation model scale sustainably as predicted by the scaling laws? Or are we far off from the scaling law predictions? What are the limits of scaling? What are the implications of the scaling laws on long-term hardware/system development?

Figures (11)

• Figure 1: Deep learning in general and deep learning based recommendation models in particular have witnessed an exponential growth in parameter size in recent years sevillaProgressMachineLearning2021mudigere2022softwarelian2021persia. Note the difference in the growth trend across different domains.
• Figure 2: Recommendation system’s performance scales with power law plus constant as we increase data size, model size and compute flops for training. (a) scaling model size through increasing MLP layers’ width. (b) scaling model size through increasing overarch layers’ width. (c) scaling model size through increasing embedding table dimensions. (d) scaling model size through increasing the number of rows in embedding tables.
• Figure 3:
• Figure 4: Data Scaling Efficiency across different model scaling schemes. While each line shows data scaling trend for a constant model size, the dashed line in each plot along with the equation, captures the pareto optimal curve. As shown, irrespective of the scaling scheme, all models have more or less the same power law scaling profile (power -0.1) when scale model and data together, implying that data scaling efficiency is the same across all model scaling schemes.
• Figure 5: Compute Scaling Efficiency -- two views: (a) scaling the amount of compute flops and dataset size in tandem (b) scaling the amount of compute flops and model size in tandem.