Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models

Jung Hwan Heo, Jeonghoon Kim, Beomseok Kwon, Byeongwook Kim, Se Jung Kwon, Dongsoo Lee

TL;DR

The paper tackles the memory bottleneck in serving large language models by introducing per-input-channel (per-IC) quantization to isolate activation outliers that hinder sub-4 bit weight quantization. It then presents AdaDim, an adaptive framework that selects between per-IC and per-OC quantization per layer by minimizing reconstruction error on a small calibration set, enabling selective and effective outlier mitigation. Empirical results show AdaDim boosts performance of weight-only quantization for both base and instruction-tuned LLMs across benchmarks such as MMLU and HumanEval, with gains up to about +4.7% and +10% respectively, and improved reconstruction error and kernel latency in several configurations. The work demonstrates that shifting the design dimension to the input channel and dynamically adapting quantization strategies can significantly improve memory-efficient LLM inference, broadening practicality for real-world deployment.

Abstract

Large Language Models (LLMs) have recently demonstrated remarkable success across various tasks. However, efficiently serving LLMs has been a challenge due to the large memory bottleneck, specifically in small batch inference settings (e.g. mobile devices). Weight-only quantization can be a promising approach, but sub-4 bit quantization remains a challenge due to large-magnitude activation outliers. To mitigate the undesirable outlier effect, we first propose per-IC quantization, a simple yet effective method that creates quantization groups within each input channel (IC) rather than the conventional per-output-channel (per-OC). Our method is motivated by the observation that activation outliers affect the input dimension of the weight matrix, so similarly grouping the weights in the IC direction can isolate outliers within a group. We also find that activation outliers do not dictate quantization difficulty, and inherent weight sensitivities also exist. With per-IC quantization as a new outlier-friendly scheme, we propose Adaptive Dimensions (AdaDim), a versatile quantization framework that can adapt to various weight sensitivity patterns. We demonstrate the effectiveness of AdaDim by augmenting prior methods such as Round-To-Nearest and GPTQ, showing significant improvements across various language modeling benchmarks for both base (up to +4.7% on MMLU) and instruction-tuned (up to +10% on HumanEval) LLMs. Code is available at https://github.com/johnheo/adadim-llm

Rethinking Channel Dimensions to Isolate Outliers for Low-bit Weight Quantization of Large Language Models

TL;DR

The paper tackles the memory bottleneck in serving large language models by introducing per-input-channel (per-IC) quantization to isolate activation outliers that hinder sub-4 bit weight quantization. It then presents AdaDim, an adaptive framework that selects between per-IC and per-OC quantization per layer by minimizing reconstruction error on a small calibration set, enabling selective and effective outlier mitigation. Empirical results show AdaDim boosts performance of weight-only quantization for both base and instruction-tuned LLMs across benchmarks such as MMLU and HumanEval, with gains up to about +4.7% and +10% respectively, and improved reconstruction error and kernel latency in several configurations. The work demonstrates that shifting the design dimension to the input channel and dynamically adapting quantization strategies can significantly improve memory-efficient LLM inference, broadening practicality for real-world deployment.

Abstract

Large Language Models (LLMs) have recently demonstrated remarkable success across various tasks. However, efficiently serving LLMs has been a challenge due to the large memory bottleneck, specifically in small batch inference settings (e.g. mobile devices). Weight-only quantization can be a promising approach, but sub-4 bit quantization remains a challenge due to large-magnitude activation outliers. To mitigate the undesirable outlier effect, we first propose per-IC quantization, a simple yet effective method that creates quantization groups within each input channel (IC) rather than the conventional per-output-channel (per-OC). Our method is motivated by the observation that activation outliers affect the input dimension of the weight matrix, so similarly grouping the weights in the IC direction can isolate outliers within a group. We also find that activation outliers do not dictate quantization difficulty, and inherent weight sensitivities also exist. With per-IC quantization as a new outlier-friendly scheme, we propose Adaptive Dimensions (AdaDim), a versatile quantization framework that can adapt to various weight sensitivity patterns. We demonstrate the effectiveness of AdaDim by augmenting prior methods such as Round-To-Nearest and GPTQ, showing significant improvements across various language modeling benchmarks for both base (up to +4.7% on MMLU) and instruction-tuned (up to +10% on HumanEval) LLMs. Code is available at https://github.com/johnheo/adadim-llm
Paper Structure (23 sections, 1 equation, 11 figures, 10 tables)

This paper contains 23 sections, 1 equation, 11 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. LLM Quantization
  4. The Activation Outlier Problem
  5. Methodology
  6. Analyzing Weight Sensitivity Patterns
  7. Per-IC quantization
  8. Adaptive Per-Channel Quantization
  9. Experiments
  10. Evaluation Setup
  11. Base Model Quantization
  12. Instruction-tuned Model Quantization
  13. Analysis
  14. Per-IC Kernel Implementation
  15. Conclusion
  16. ...and 8 more sections

Figures (11)

  • Figure 1: Per-input-channel quantization. Activation outliers that affect certain input channels (ICs) amplify quantization errors. Such sensitive ICs exist in all groups in the conventional per-output-channel (per-OC) quantization. With our per-IC scheme, the outlier effect is isolated.
  • Figure 2: Weight sensitivity patterns of LLaMA-V2-7B. Darker colors indicate larger activation magnitude (red), and higher weight sensitivity (blue). Sensitivity is computed with fisher information approximated by the squared of the gradient. We downsample the grids with a 16x16 maxpool kernel and take the log scale for clarity. Left: The presence of activation outliers lead to sensitive rows. Right: Mild activations lead to less sensitive rows, while the dominant sensitivity dimension (row or column) can change across network depth even for the same module.
  • Figure 3: Base model results. Evaluating the effectiveness of our AdaDim framework for LLaMA base models on Massive Multi-task Language Understanding (MMLU) and commonsense reasoning (CSR) tasks. Performance boosts (in green) indicate additional gains from adaptively switching to per-IC quantization. We observe notable gains over the original per-OC versions of RTN and GPTQ frantar2022gptq, often matching or even surpassing AWQ lin2023awq.
  • Figure 4: Sweeping various quantization configurations for LLaMA-V2-7B. Average bits per weight increases from left to right (x-axis). AdaDim can further close the gap between INT3/4 and FP16.
  • Figure 5: Adaptive dimension selection. By adaptively switching to per-IC quantization, AdaDim can reduce the reconstruction error up to 6$\times$ in the RTN setting. The decisions vary across model size (7B vs. 70B) and task (language modeling vs. math), showcasing the versatility of AdaDim.
  • ...and 6 more figures