Effect of Weight Quantization on Learning Models by Typical Case Analysis

TL;DR

This work tackles how weight quantization hyperparameters $b$ and $\omega$ affect generalization in high-dimensional learning. It employs the replica method with a replica-symmetric (RS) analysis, extended to replica-symmetry-breaking (RSB) regimes, and validates predictions with an approximate message-passing (AMP) algorithm. Key contributions include phase diagrams showing RS and RSB regions, the existence of an optimal quantization width $\omega$ that minimizes generalization error, and the finding that non-uniform quantization improves stability and delays overparameterization under quantization. The results provide principled guidance for selecting quantization hyperparameters to balance accuracy and computational efficiency on resource-constrained devices, and offer a rigorous framework linking statistical physics methods to quantized regression in the high-dimensional limit.

Abstract

This paper examines the quantization methods used in large-scale data analysis models and their hyperparameter choices. The recent surge in data analysis scale has significantly increased computational resource requirements. To address this, quantizing model weights has become a prevalent practice in data analysis applications such as deep learning. Quantization is particularly vital for deploying large models on devices with limited computational resources. However, the selection of quantization hyperparameters, like the number of bits and value range for weight quantization, remains an underexplored area. In this study, we employ the typical case analysis from statistical physics, specifically the replica method, to explore the impact of hyperparameters on the quantization of simple learning models. Our analysis yields three key findings: (i) an unstable hyperparameter phase, known as replica symmetry breaking, occurs with a small number of bits and a large quantization width; (ii) there is an optimal quantization width that minimizes error; and (iii) quantization delays the onset of overparameterization, helping to mitigate overfitting as indicated by the double descent phenomenon. We also discover that non-uniform quantization can enhance stability. Additionally, we develop an approximate message-passing algorithm to validate our theoretical results.

Figures (10)

• Figure 1: (a) and (b): Quanzied values $\widehat{w}=\varphi(w)$ as a function of continuous parameter $w$ at $\omega=8$ and $n_p=6$ for (a) uniform and (b) non-uniform cases. The diagonal lines represent the identity map. (c) and (d): Loss function at $N=M=1$ for $y=0$ and $x=1$ with $\lambda=0$ for (c) uniform and (d) non-uniform quantization corresponding to (a) and (b), respectively.
• Figure 2: Phase diagrams on $b-\omega$ plane for quantized regression at $(\sigma, \alpha)=(0.01,1.5)$. Shaded and blighted regions are RS and RSB phase, respectively. (a) and (b) are under uniform quantization at $\lambda = 0.0$ and $\lambda = 1.0$, respectively. (c) and (d) are under non-uniform quantization at $\lambda = 0.0$ and $\lambda = 1.0$, respectively.
• Figure 3: Comparison between the distribution of continuous value $hz\slash\widehat{\Theta}$ (solid line) and quantized values in $\Omega$ with $\omega=10$ (dots) for (a) $n_p=2$ and (b) $n_p=3$ under uniform quantization. Here, we set $h\slash\widehat{\Theta}=1$ for simplicity, and areas corresponding to the discrete values are shaded separately.
• Figure 4: Expected generalization error under RS assumption as a function of $\omega$ for (a) $(\sigma, \lambda, \alpha)=(0.01, 0.01, 1.4)$ and (b) $(\sigma, \lambda, \alpha)=(1.0, 0.01, 1.4)$. Solid lines and dashesd lines represent the result of uniform and non-uniform quantization, respectively. RSB phase is indicated by black arrows.
• Figure 5: Expected generalization error under RS assumption as a function of $\alpha = N/M$ for (a) uniform quantization and (b) non-uniform quantization. The dashed lines represent the result of ridge regression.