Pyramid Vector Quantization for LLMs

TL;DR

This work tackles the challenge of compressing large language models by quantizing weights on the unit sphere using Pyramid Vector Quantization (PVQ), which provides a memory-efficient implicit codebook and a search-free encoding/decoding workflow. The authors augment PVQ with coherence processing via Hadamard rotations, Hessian-informed amplitude handling, and Beta-distribution-based amplitude quantization to minimize quantization error while achieving flexible bits-per-weight configurations, including non-integer values. They demonstrate state-of-the-art results across several LLM families, achieving as low as $3.25$ BPW with minimal accuracy loss on downstream tasks, and provide CUDA-accelerated PVQ kernels to enable practical deployment. The approach shows strong performance for both weight-only and weight-plus-activation quantization, highlighting a scalable, hardware-friendly path to post-training and potential train-time quantization of LLMs. The work thus offers a principled, geometry-aware quantization framework with broad implications for efficient deployment of large neural models.

Abstract

Recent works on compression of large language models (LLM) using quantization considered reparameterizing the architecture such that weights are distributed on the sphere. This demonstratively improves the ability to quantize by increasing the mathematical notion of coherence, resulting in fewer weight outliers without affecting the network output. In this work, we aim to further exploit this spherical geometry of the weights when performing quantization by considering Pyramid Vector Quantization (PVQ) for large language models. Arranging points evenly on the sphere is notoriously difficult, especially in high dimensions, and in case approximate solutions exists, representing points explicitly in a codebook is typically not feasible due to its additional memory cost. Instead, PVQ uses a fixed integer lattice on the sphere by projecting points onto the 1-sphere, which allows for efficient encoding and decoding without requiring an explicit codebook in memory. To obtain a practical algorithm, we propose to combine PVQ with scale quantization for which we derive theoretically optimal quantizations, under empirically verified assumptions. Further, we extend pyramid vector quantization to use Hessian information to minimize quantization error under expected feature activations, instead of only relying on weight magnitudes. Experimentally, we achieves state-of-the-art quantization performance with pareto-optimal trade-off between performance and bits per weight and bits per activation, compared to compared methods. On weight-only, we find that we can quantize a Llama-3 70B model to 3.25 bits per weight and retain 98\% accuracy on downstream tasks.

Figures (13)

• Figure 1: Illustration of the PVQ integer lattice in $d=3$ dimensions with increasing pulses $k$ from 1 to 6. Points on the pyramid ${\mathcal{P}}_{3, k}$ are projected onto the sphere ${\mathcal{S}}_{3, k}$.
• Figure 2: Illustration of points on pyramid ${\mathcal{P}}_{2, 7}$, their projections onto the sphere ${\mathcal{S}}_{2, 7}$ and codes in ${\mathcal{C}}_{2,7}$.
• Figure 3: Signal-to-quantization-noise-ratio (QSNR) of quantization methods on standard Gaussian source. PVQ achieves high QSNR close to E8, which uses an optimal packing on a uniform source. PVQ uses an implicit codebook and is search-free, thereby amenable to quantization of weight and activations.
• Figure 4: Effect of different direction and amplitude bits on effective PVQ quantization grid. The distribution automatically matches the theoretical $\text{Beta}$ weight distribution.
• Figure 5: Theoretical Beta distribution of \ref{['thm:beta-dist']} closely match amplitudes of rotated weights in trained LLMs, here demonstrated for empirical weight distributions of a pretrained Llama-v3 8B.

Theorems & Definitions (1)

• Theorem 3.1