Only relative ranks matter in weight-clustered large language models

Abstract

Large language models (LLMs) contain billions of parameters, yet many exact values are not essential. We show that what matters most is the relative rank of weights-whether one connection is stronger or weaker than another-rather than precise magnitudes. To reduce the number of unique weight values, we apply weight clustering to pretrained models, replacing every weight matrix with K shared values from K-means. For Llama 3.1-8B-Instruct and SmolLM2-135M, reducing each matrix to only 16-64 distinct values preserves strong accuracy without retraining, providing a simple, training-free method to compress LLMs on disk. Optionally fine-tuning only the cluster means (centroids) recovers 30-40 percent of the remaining accuracy gap at minimal cost. We then systematically randomize cluster means while keeping assignments fixed. Scrambling the relative ranks of the clusters degrades quality sharply-perplexity can increase by orders of magnitude-even when global statistics such as mean and variance are preserved. In contrast, rank-preserving randomizations cause almost no loss at mid and late layers. On the other hand, when many layers are perturbed simultaneously, progressive layer-by-layer replacement reveals that scale drift-not rank distortion-is the dominant collapse mechanism; however, an affine correction w' = aw + b with a > 0 (which preserves both rank order and overall weight distribution) can substantially delay this drift. This rank-based perspective offers a new lens on model compression and robustness.

Figures (3)

• Figure 1: (a) A neural network layer showing connections between input neurons (top) and output neurons (bottom). Edge colors indicate the magnitude of the weights. (b) A clustered version of the same weight matrix with $K=2$ clusters. In this paper, we demonstrate that when the clustered matrix remains performant, only the relative clustering of the weights (here, cold vs. hot) affects model accuracy. The absolute scale of each cluster can be changed by an affine transformation without substantially degrading model performance.
• Figure 2: Downstream task accuracy comparison for clustered versions of Llama 3.1-8B-Instruct and our compressed Llama 3B model. We see that weight clustering largely preserves accuracy across all tasks without requiring any fine-tuning.
• Figure 3: Normalized perplexity (PPL / baseline) as centroids are progressively replaced with rank-preserving-random values, one transformer block at a time from deepest to shallowest. Thick lines with markers show affine-corrected replacement (mean and variance of the reconstructed weights are preserved); thin lines of matching color show uncorrected replacement. Each curve normalizes by its own baseline (Llama 3.1-8B-Instruct: 9.08; CAI pre-compressed Llama 3B: 12.32; SmolLM2-135M: 27.52).