Compressibility Measures Complexity: Minimum Description Length Meets Singular Learning Theory

TL;DR

This work develops a singular MDL framework to quantify neural network complexity through compressibility, using the local learning coefficient (LLC) from singular learning theory. It derives that the asymptotic two-part code redundancy satisfies $R_n = \lambda \log n - (m-1) \log \log n + O_p(1)$, linking LLC to model complexity via degeneracy in the loss landscape. Empirically, it shows a strong, often linear, relationship between LLC estimates and compression limits (notably under quantization) across Pythia transformers up to $6.9$B parameters, validating LLC as a principled proxy for complexity in large models. The results offer a rigorous pathway to evaluate the fundamental limits of model compression and guide the development of compression techniques aligned with theoretical limits.

Abstract

We study neural network compressibility by using singular learning theory to extend the minimum description length (MDL) principle to singular models like neural networks. Through extensive experiments on the Pythia suite with quantization, factorization, and other compression techniques, we find that complexity estimates based on the local learning coefficient (LLC) are closely, and in some cases, linearly correlated with compressibility. Our results provide a path toward rigorously evaluating the limits of model compression.

Figures (21)

• Figure 1: Loss landscape volume determines model compressibility. We generalize the minimum description length (MDL) principle to singular models like neural networks, which exhibit redundancy in their loss landscape (right two panels). This redundancy, or "degeneracy", is the leading order contribution to model compressibility --- not the curvature as determined by the Hessian (left two panels). From left to right: (1) regular model with symmetric quadratic loss; (2) regular model with elliptical quadratic loss showing different curvatures along principal axes; (3) minimally singular model with a redundant parameter, creating a valley of degenerate optima; (4) "normal-crossing" singularity showing higher order degeneracy.
• Figure 2: Degeneracy determines volume and compressibility Left: The relationship between error tolerance ($\epsilon$) and the number of bits required to encode parameters for three different model geometries shows that the volume-scaling exponent $\lambda$ ("local learning coefficient," LLC) determines compressibility. Right three panels show level set contours of the respective loss landscapes at $\epsilon=2,\dots,2^{-8}$ (same as in \ref{['fig:smdl-pedagogical']}). 1. Regular: A model with elliptical level sets requiring approximately $d/2 \log(1/\epsilon)$ bits ($\lambda=1$ for $d=2$) to specify a point within tolerance $\epsilon$. 2. Minimally Singular: A model with free parameter, requiring only $(d-1)/2 \log(1/\epsilon)$ bits ($\lambda=1/2$ for $d=2$) due to degeneracy along the $w_2$ direction. 3. Singular: A model exhibiting a more complex geometric structure requiring approximately $\lambda \log(1/\epsilon)$ bits, with $\lambda = 1/4$.
• Figure 3: Sensitivity of Pythia models of different sizes to quantization. On the left, we show the loss increase as a function of $n_q$, with the black dashed line indicating our choice of loss tolerance $\epsilon=0.5$. On the right, we show the critical $n_q$, i.e., the $n_q$ for which $\Delta \text{Loss}=\epsilon$ as a function of the LLC for different training checkpoints of Pythia models. The transparent points are not included in the linear fits. The checkpoints increase with the LLC, that is, training time moves from left to right along all curves.
• Figure 4: Sensitivity of Pythia models to Quantization. The panels show loss increase as a function of $n_q$, with the lower right panel showing the critical $n_q$ as a function of LLC for $\epsilon = 0.5$. Note that for Pythia-14M and Pythia-70M, checkpoint 90k is not included since our quantization algorithm fails for these checkpoints. We do a linear fit excluding the transparent points, finding $R^2$ values of 0.98 and above.
• Figure 5: Critical $\mathbf{n_q}$ vs. LLC for different $\boldsymbol{\epsilon}$ values. We see that the curves are qualitatively $\epsilon$-insensitive. Note that for Pythia-14M and Pythia-70M, checkpoint 90k is not included since our quantization algorithm fails for these checkpoints.

Theorems & Definitions (15)

• Theorem 1
• Theorem 2: watanabeAlgebraicGeometryStatistical2009Arnold2012-cq
• Definition 1
• Lemma 1
• proof
• Lemma 2
• proof
• Lemma 3
• proof
• Theorem 3