CRVQ: Channel-Relaxed Vector Quantization for Extreme Compression of LLMs

TL;DR

CRVQ tackles extreme compression of LLMs by introducing channel-aware vector quantization. It selects a small subset of critical weight channels, reorders them, and applies extended codebooks to those channels via additive VQ, enabling near 1-bit quantization with minimal bit-width overhead. Across multiple models (1.3B–13B) and baselines, CRVQ delivers substantial perplexity reductions and improved zero-shot accuracy, outperforming strong PTQ methods while maintaining hardware efficiency. The approach is hardware-friendly, flexible in codebook design, and demonstrates robust performance gains with practical deployment guidance.

Abstract

Powerful large language models (LLMs) are increasingly expected to be deployed with lower computational costs, enabling their capabilities on resource-constrained devices. Post-training quantization (PTQ) has emerged as a star approach to achieve this ambition, with best methods compressing weights to less than 2 bit on average. In this paper, we propose Channel-Relaxed Vector Quantization (CRVQ), a novel technique that significantly improves the performance of PTQ baselines at the cost of only minimal additional bits. This state-of-the-art extreme compression method achieves its results through two key innovations: (1) carefully selecting and reordering a very small subset of critical weight channels, and (2) leveraging extended codebooks to relax the constraint of critical channels. With our method, we demonstrate a 38.9\% improvement over the current strongest sub-2-bit PTQ baseline, enabling nearer lossless 1-bit compression. Furthermore, our approach offers flexible customization of quantization bit-width and performance, providing a wider range of deployment options for diverse hardware platforms.

Figures (7)

• Figure 1: Comparison of different quantization methods. Uniform quantization treats each weight to be quantized as a scalar, whereas vector quantization considers weight segments as vectors. Our proposed CRVQ introduces distinct importance across weight channels, with the more critical channels selected and highlighted in dark colors. We use additional extended codebooks to quantize the vectors formed by these critical channels.
• Figure 2: Visualization of Equation \ref{['eq:bitwidth']}. The curves represent bit-width based on the vector dimension $d$ and the codebook bit-width $e$. The top (a) and bottom (b) plots correspond to $m=1$ and $m=2$, respectively.
• Figure 3: Illustration of the proposed CRVQ, consisting of three phases. On the left, channels are reordered by importance, with circle shading indicating weight significance. Next, vectors are fitted to codebooks, where critical weights are represented as the sum of corresponding vectors from all codebooks (1 to 4). The right sub-figure shows channels reordered to their original sequence.
• Figure 4: Quantization performance varies with the critical channel ratio $\lambda$ and codebook count $m$.
• Figure 5: Effectiveness of CRVQ when combine with stronger basic codebook, i.e. $e$ changes from 8 to 10 and 12. The performance consistently improves as the extended codebook count increases.