Low-Bit Quantization Favors Undertrained LLMs: Scaling Laws for Quantized LLMs with 100T Training Tokens

Abstract

We reveal that low-bit quantization favors undertrained large language models (LLMs) by observing that models with larger sizes or fewer training tokens experience less quantization-induced degradation (QiD) when applying low-bit quantization, whereas smaller models with extensive training tokens suffer significant QiD. To gain deeper insights into this trend, we study over 1500 quantized LLM checkpoints of various sizes and at different training levels (undertrained or fully trained) in a controlled setting, deriving scaling laws for understanding the relationship between QiD and factors such as the number of training tokens, model size and bit width. With the derived scaling laws, we propose a novel perspective that we can use QiD to measure an LLM's training levels and determine the number of training tokens required for fully training LLMs of various sizes. Moreover, we use the scaling laws to predict the quantization performance of different-sized LLMs trained with 100 trillion tokens. Our projection shows that the low-bit quantization performance of future models, which are expected to be trained with over 100 trillion tokens, may NOT be desirable. This poses a potential challenge for low-bit quantization in the future and highlights the need for awareness of a model's training level when evaluating low-bit quantization research. To facilitate future research on this problem, we release all the 1500+ quantized checkpoints used in this work at https://huggingface.co/Xu-Ouyang.

Figures (14)

• Figure 1: Scaling laws for predicting Quantization-induced Degradation (QiD, denoted as $\Delta_qLoss$) in 7B, 70B, and 405B models trained on up to 100 trillion ($10^{14}$) tokens. While low-bit quantization yields acceptable QiD for undertrained LLMs (trained with $\leq 10^{12}$ tokens), it is predicted to become undesirable when applied to fully trained LLMs (e.g., trained with 100 trillion tokens, a milestone expected to be reached in the next few years), particularly for smaller models. Note that the gray areas in this figure indicate levels of QiD that degrade the model's predictions to a level worse than random guessing.
• Figure 2: Performance of LLMs after low-bit quantization at different sizes and training levels. It is obvious that the models which are smaller or trained with more tokens suffer from greater quantization-induced degradation.
• Figure 3: The fitted scaling law of QiD with respect to the number of training tokens in the form of Eq (\ref{['eq:data']}), where $\beta$ is fitted to be 0.5316.
• Figure 4: The fitted scaling law of QiD with respect to the model size (i.e., the number of non-embedding parameters) in the form of Eq (\ref{['eq:params']}), where $\alpha$ is fitted to be 0.2276.
• Figure 5: The fitted scaling law of QiD with respect to the bit width in the form of Eq (\ref{['eq:bit']}), where $\gamma$ is fitted to be 5.4812.