Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Qrazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring

Dongyoung Lee, Seungkyu Choi, Ik Joon Chang

TL;DR

QRazor tackles the challenge of deploying large language models with limited memory and compute by delivering a reliable 4-bit quantization pipeline without fine-tuning. It first quantizes weights, activations, and KV caches to a high-bit base (W8A16 or W8A16KV8) using absmax scaling, preserving accuracy close to FP16, then compresses to 4-bit with Significant Data Razoring (SDR) that retains only the most salient bits. The method is complemented by a decompression-free integer arithmetic unit, enabling direct low-precision computation with reduced hardware cost. Empirical results across LLaMA-2/3 and Mistral-7B show strong zero-shot performance and competitiveness against state-of-the-art 4-bit quantization, while hardware simulations report substantial area and power savings, making QRazor practical for edge and datacenter deployments alike.

Abstract

Large-scale language models (LLMs) excel in language processing tasks but face deployment challenges due to high memory and computational demands. While low-bit quantization, such as 4-bit techniques, offers a potential solution, these methods often suffer from significant accuracy loss or require considerable effort for implementation such as reordering, rotation, etc. To address these challenges, we propose QRazor, a simple yet effective quantization scheme that enables 4-bit quantization of weights, activations, and KV cache in transformer-based LLMs. QRazor operates in two stages: first, quantizing data using 8 or 16-bit integers as a basis with absolute max scaling to preserve accuracy close to full-precision models, and second, compressing the quantized data to 4-bit using our significant data razoring (SDR) technique, which retains only the four most salient bits. Without any additional requirment of fine-tuning or additional training, QRazor achieves performance similar or better compared to state-of-the-art in 4-bit quantization method, surpassing Smoothquant and QLLM by over 12 points and Quarot(RTN) by more than 2.9 points in zero-shot reasoning task accuracy on the LLaMA2-7B model. Additionally, we introduce an integer-based arithmetic unit optimized for QRazor, allowing direct low-precision operations on SDR data without decompression.

Qrazor: Reliable and Effortless 4-bit LLM Quantization by Significant Data Razoring

TL;DR

QRazor tackles the challenge of deploying large language models with limited memory and compute by delivering a reliable 4-bit quantization pipeline without fine-tuning. It first quantizes weights, activations, and KV caches to a high-bit base (W8A16 or W8A16KV8) using absmax scaling, preserving accuracy close to FP16, then compresses to 4-bit with Significant Data Razoring (SDR) that retains only the most salient bits. The method is complemented by a decompression-free integer arithmetic unit, enabling direct low-precision computation with reduced hardware cost. Empirical results across LLaMA-2/3 and Mistral-7B show strong zero-shot performance and competitiveness against state-of-the-art 4-bit quantization, while hardware simulations report substantial area and power savings, making QRazor practical for edge and datacenter deployments alike.

Abstract

Large-scale language models (LLMs) excel in language processing tasks but face deployment challenges due to high memory and computational demands. While low-bit quantization, such as 4-bit techniques, offers a potential solution, these methods often suffer from significant accuracy loss or require considerable effort for implementation such as reordering, rotation, etc. To address these challenges, we propose QRazor, a simple yet effective quantization scheme that enables 4-bit quantization of weights, activations, and KV cache in transformer-based LLMs. QRazor operates in two stages: first, quantizing data using 8 or 16-bit integers as a basis with absolute max scaling to preserve accuracy close to full-precision models, and second, compressing the quantized data to 4-bit using our significant data razoring (SDR) technique, which retains only the four most salient bits. Without any additional requirment of fine-tuning or additional training, QRazor achieves performance similar or better compared to state-of-the-art in 4-bit quantization method, surpassing Smoothquant and QLLM by over 12 points and Quarot(RTN) by more than 2.9 points in zero-shot reasoning task accuracy on the LLaMA2-7B model. Additionally, we introduce an integer-based arithmetic unit optimized for QRazor, allowing direct low-precision operations on SDR data without decompression.
Paper Structure (21 sections, 5 figures, 10 tables)

This paper contains 21 sections, 5 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. The QRazor Scheme
  5. Quantization Stage: W8A16 or W8A16KV8
  6. Compression Stage: Significant Data Razoring
  7. Decompression-Free Arithmetic Operation
  8. Experiments
  9. Setup
  10. LLM Results
  11. Ablation Studies
  12. Hardware Efficiency
  13. Conclusion
  14. Appendix
  15. Importance of the Weight Quantization
  16. ...and 6 more sections

Figures (5)

  • Figure 1: The overall concept of our QRazor scheme.
  • Figure 2: Leading '1' position before the compression for (a) activation and query, and (b) key. (c) portion of zeroed elements between "Before 4-bit compression" and "After 4-bit compression"
  • Figure 3: The concept of a commonly perceived decompression method(a) compared with the proposed decompression-free arithmetic(b).
  • Figure 4: SDR Encoding Scheme Consisting of Bitwise OR Operations for detecting the razoring point.
  • Figure 5: The QRazor quantization flow in attention and feed-forward network layers