Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

SDQ: Sparse Decomposed Quantization for LLM Inference

Geonhwa Jeong, Po-An Tsai, Stephen W. Keckler, Tushar Krishna

TL;DR

SDQ tackles the compute and memory bottlenecks of large language model inference by fusing structured sparsity with low-bit quantization in a post-training framework. It decomposes weight tensors into outlier and inlier components, applying differential quantization to each to match hardware—particularly structured sparse tensor cores—while preserving accuracy. Across OPT and LLaMA families, SDQ demonstrates substantial throughput gains with minimal quality degradation, outperforming sparsification-only and weight-quantization-only baselines. The method is designed to be orthogonal to existing compression techniques and suitable for future sparse-tensor accelerators, offering a practical pathway to efficient LLM deployment.

Abstract

Recently, large language models (LLMs) have shown surprising performance in task-specific workloads as well as general tasks with the given prompts. However, to achieve unprecedented performance, recent LLMs use billions to trillions of parameters, which hinder the wide adaptation of those models due to their extremely large compute and memory requirements. To resolve the issue, various model compression methods are being actively investigated. In this work, we propose SDQ (Sparse Decomposed Quantization) to exploit both structured sparsity and quantization to achieve both high compute and memory efficiency. From our evaluations, we observe that SDQ can achieve 4x effective compute throughput with <1% quality drop.

SDQ: Sparse Decomposed Quantization for LLM Inference

TL;DR

SDQ tackles the compute and memory bottlenecks of large language model inference by fusing structured sparsity with low-bit quantization in a post-training framework. It decomposes weight tensors into outlier and inlier components, applying differential quantization to each to match hardware—particularly structured sparse tensor cores—while preserving accuracy. Across OPT and LLaMA families, SDQ demonstrates substantial throughput gains with minimal quality degradation, outperforming sparsification-only and weight-quantization-only baselines. The method is designed to be orthogonal to existing compression techniques and suitable for future sparse-tensor accelerators, offering a practical pathway to efficient LLM deployment.

Abstract

Recently, large language models (LLMs) have shown surprising performance in task-specific workloads as well as general tasks with the given prompts. However, to achieve unprecedented performance, recent LLMs use billions to trillions of parameters, which hinder the wide adaptation of those models due to their extremely large compute and memory requirements. To resolve the issue, various model compression methods are being actively investigated. In this work, we propose SDQ (Sparse Decomposed Quantization) to exploit both structured sparsity and quantization to achieve both high compute and memory efficiency. From our evaluations, we observe that SDQ can achieve 4x effective compute throughput with <1% quality drop.
Paper Structure (19 sections, 11 figures, 4 tables)

This paper contains 19 sections, 11 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Compressing LLM models
  4. Sparification
  5. Quantization
  6. Comparing Compression Methods
  7. Effective compute throughput for sparsification
  8. Effective compute throughput for quantization
  9. Average bits per weight element
  10. SDQ: Tackling Challenges in LLM Compression with a Hybrid method
  11. SDQ For LLMs
  12. LLM inference performance
  13. Evaluation
  14. Methodology
  15. Perplexity evaluation
  16. ...and 4 more sections

Figures (11)

  • Figure 1: Effective compute throughput and perplexity increase comparison of sparsification-only, quantization-only, and SDQ on OPT-6.7B and LLaMA-7B.
  • Figure 2: A transformer block architecture.
  • Figure 3: Overview of sparsification and quantization.
  • Figure 4: Data size for 32 elements with 1:4/2:4/3:4/Dense sparsity with quantization using 1) 32 bit scale factor with Q-Vector size of 16 (first row) 2) 8 bit scale factor with Q-Vector size of 32 (second row). SF and Q-VS represent Scale Factor and Q-Vector Size.
  • Figure 5: Local outlier extraction using N:8 for global outliers and semi-local outliers.
  • ...and 6 more figures