Densing Law of LLMs
Chaojun Xiao, Jie Cai, Weilin Zhao, Guoyang Zeng, Biyuan Lin, Jie Zhou, Zhi Zheng, Xu Han, Zhiyuan Liu, Maosong Sun
TL;DR
The paper defines capability density as the ratio of an LLM's effective parameter size to its actual size and develops a two-step loss-then-performance framework to estimate this quantity from reference-model scaling laws. It demonstrates an empirical Densing Law: capacity density grows exponentially over time, with density roughly doubling every three months, implying substantial, sustained reductions in inference costs for equivalent performance. Through evaluation on multiple open-source base LLMs and benchmarks, the work argues for a density-centric view of LLM development and advocates density-optimal training over sheer scale to achieve sustainable efficiency. This paradigm connects scaling behavior with hardware and algorithmic advances, offering a practical roadmap for building high-performing, cost-efficient models.
Abstract
Large Language Models (LLMs) have emerged as a milestone in artificial intelligence, and their performance can improve as the model size increases. However, this scaling brings great challenges to training and inference efficiency, particularly for deploying LLMs in resource-constrained environments, and the scaling trend is becoming increasingly unsustainable. This paper introduces the concept of ``\textit{capacity density}'' as a new metric to evaluate the quality of the LLMs across different scales and describes the trend of LLMs in terms of both effectiveness and efficiency. To calculate the capacity density of a given target LLM, we first introduce a set of reference models and develop a scaling law to predict the downstream performance of these reference models based on their parameter sizes. We then define the \textit{effective parameter size} of the target LLM as the parameter size required by a reference model to achieve equivalent performance, and formalize the capacity density as the ratio of the effective parameter size to the actual parameter size of the target LLM. Capacity density provides a unified framework for assessing both model effectiveness and efficiency. Our further analysis of recent open-source base LLMs reveals an empirical law (the densing law)that the capacity density of LLMs grows exponentially over time. More specifically, using some widely used benchmarks for evaluation, the capacity density of LLMs doubles approximately every three months. The law provides new perspectives to guide future LLM development, emphasizing the importance of improving capacity density to achieve optimal results with minimal computational overhead.