A Latent Variable Framework for Scaling Laws in Large Language Models
Peiyao Cai, Chengyu Cui, Felipe Maia Polo, Seamus Somerstep, Leshem Choshen, Mikhail Yurochkin, Moulinath Banerjee, Yuekai Sun, Kean Ming Tan, Gongjun Xu
TL;DR
The paper tackles the challenge that classical LLM scaling laws, which relate performance to model size and data, poorly capture the heterogeneous, multi-faceted capabilities across diverse model families and benchmarks. It introduces a latent-variable framework where family-specific latent abilities generate benchmark performance through a beta-based IRT-like model, enabling coherent cross-family evaluation, uncertainty quantification, and interpretable skill dimensions. The authors develop a likelihood-based estimator with identifiability constraints, prove consistency and asymptotic normality, and provide scalable computation via projected stochastic gradient ascent and posterior sampling. Empirically, they fit the model to 12 Open LLM Leaderboard benchmarks, uncover four interpretable latent skills, generate prediction intervals for unevaluated models, and derive compute-optimal scaling guidance for different skills. This framework offers principled tooling for planning compute, benchmarking, and understanding how design choices shape emergent capabilities in large language models.
Abstract
We propose a statistical framework built on latent variable modeling for scaling laws of large language models (LLMs). Our work is motivated by the rapid emergence of numerous new LLM families with distinct architectures and training strategies, evaluated on an increasing number of benchmarks. This heterogeneity makes a single global scaling curve inadequate for capturing how performance varies across families and benchmarks. To address this, we propose a latent variable modeling framework in which each LLM family is associated with a latent variable that captures the common underlying features in that family. An LLM's performance on different benchmarks is then driven by its latent skills, which are jointly determined by the latent variable and the model's own observable features. We develop an estimation procedure for this latent variable model and establish its statistical properties. We also design efficient numerical algorithms that support estimation and various downstream tasks. Empirically, we evaluate the approach on 12 widely used benchmarks from the Open LLM Leaderboard (v1/v2).