Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression

Xi Zhang, Xiaolin Wu, Jiamang Wang, Weisi Lin

TL;DR

GLVQ tackles the challenge of deploying large language models under tight resource constraints by introducing grouped lattice vector quantization with per-group learnable lattices, group-specific companding, and salience-driven bit allocation. The method jointly optimizes a group generation matrix $\mathbf{G}_{g}$ and companding parameters while using Babai rounding for stable training, enabling efficient runtime decoding as a matrix-vector product. Across Llama 1/2 models, GLVQ yields superior perplexity and robust zero-shot accuracy at ultra-low bit-widths (including fractional and sub-2-bit regimes), with favorable throughput and memory footprints. This approach offers a practical path to high-fidelity LLM inference on resource-constrained hardware, with potential extensions to hierarchical grouping and activations.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities but typically require extensive computational resources and memory for inference. Post-training quantization (PTQ) can effectively reduce these demands by storing weights in lower bit-width formats. However, standard uniform quantization often leads to notable performance degradation, particularly in low-bit scenarios. In this work, we introduce a Grouped Lattice Vector Quantization (GLVQ) framework that assigns each group of weights a customized lattice codebook, defined by a learnable generation matrix. To address the non-differentiability of the quantization process, we adopt Babai rounding to approximate nearest-lattice-point search during training, which enables stable optimization of the generation matrices. Once trained, decoding reduces to a simple matrix-vector multiplication, yielding an efficient and practical quantization pipeline. Experiments on multiple benchmarks show that our approach achieves a better trade-off between model size and accuracy compared to existing post-training quantization baselines, highlighting its effectiveness in deploying large models under stringent resource constraints. Our source code is available on GitHub repository: https://github.com/xzhang9308/GLVQ.

Learning Grouped Lattice Vector Quantizers for Low-Bit LLM Compression

TL;DR

GLVQ tackles the challenge of deploying large language models under tight resource constraints by introducing grouped lattice vector quantization with per-group learnable lattices, group-specific companding, and salience-driven bit allocation. The method jointly optimizes a group generation matrix and companding parameters while using Babai rounding for stable training, enabling efficient runtime decoding as a matrix-vector product. Across Llama 1/2 models, GLVQ yields superior perplexity and robust zero-shot accuracy at ultra-low bit-widths (including fractional and sub-2-bit regimes), with favorable throughput and memory footprints. This approach offers a practical path to high-fidelity LLM inference on resource-constrained hardware, with potential extensions to hierarchical grouping and activations.

Abstract

Large Language Models (LLMs) have demonstrated remarkable capabilities but typically require extensive computational resources and memory for inference. Post-training quantization (PTQ) can effectively reduce these demands by storing weights in lower bit-width formats. However, standard uniform quantization often leads to notable performance degradation, particularly in low-bit scenarios. In this work, we introduce a Grouped Lattice Vector Quantization (GLVQ) framework that assigns each group of weights a customized lattice codebook, defined by a learnable generation matrix. To address the non-differentiability of the quantization process, we adopt Babai rounding to approximate nearest-lattice-point search during training, which enables stable optimization of the generation matrices. Once trained, decoding reduces to a simple matrix-vector multiplication, yielding an efficient and practical quantization pipeline. Experiments on multiple benchmarks show that our approach achieves a better trade-off between model size and accuracy compared to existing post-training quantization baselines, highlighting its effectiveness in deploying large models under stringent resource constraints. Our source code is available on GitHub repository: https://github.com/xzhang9308/GLVQ.
Paper Structure (35 sections, 1 theorem, 29 equations, 1 figure, 13 tables, 1 algorithm)

This paper contains 35 sections, 1 theorem, 29 equations, 1 figure, 13 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. LLM Post-Training Quantization (PTQ)
  4. Lattice Vector Quantization (LVQ)
  5. Companding for Quantization
  6. Method
  7. Salience-Determined Bit Allocation
  8. Lattice Codebook Learning
  9. Group-Specific Companding
  10. Quantization Pipeline and Runtime Decoding
  11. Experiments
  12. Perplexity Results on Llama Models
  13. Zero-Shot Accuracy on Llama Models
  14. Fractional and Sub-2-Bit Quantization
  15. Inference Throughput
  16. ...and 20 more sections

Key Result

Lemma A.1

For any $1 \leq j \leq n$, the coefficient $\alpha_j$ satisfies:

Figures (1)

  • Figure 1: An overview of the GLVQ framework. The LLM weight matrix is partitioned into column-wise groups, each assigned a salience-determined bitwidth. Each group undergoes a tailored companding transformation and is quantized using a learned group-specific lattice codebook, enabling fine-grained control over precision and compression efficiency.

Theorems & Definitions (1)

  • Lemma A.1