Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Post-Training Quantization via Future Activation Awareness

Zheqi Lv, Zhenxuan Fan, Qi Tian, Wenqiao Zhang, Yueting Zhuang

TL;DR

PTQ for large language models often suffers quantization bias and error accumulation because per-layer scales rely on current activations, a problem amplified when calibration data do not match deployment data. The authors propose Future-Aware Quantization (FAQ), which uses future-layer activations to guide per-layer scales, augmented by a window-wise preview and a pre-searched configuration to balance accuracy and overhead; a theoretical bound suggests reduced quantization error compared with AWQ. Empirically, FAQ yields consistent improvements in perplexity and zero-shot accuracy across diverse models (e.g., Qwen, LLaMA families), with larger gains at ultra-low bit-widths (3-bit) and negligible computational overhead. This approach enables robust, edge-friendly deployment of LLMs by mitigating calibration bias and minimizing the need for backpropagation or data reconstruction during quantization.

Abstract

Post-training quantization (PTQ) is a widely used method to compress large language models (LLMs) without fine-tuning. It typically sets quantization hyperparameters (e.g., scaling factors) based on current-layer activations. Although this method is efficient, it suffers from quantization bias and error accumulation, resulting in suboptimal and unstable quantization, especially when the calibration data is biased. To overcome these issues, we propose Future-Aware Quantization (FAQ), which leverages future-layer activations to guide quantization. This allows better identification and preservation of important weights, while reducing sensitivity to calibration noise. We further introduce a window-wise preview mechanism to softly aggregate multiple future-layer activations, mitigating over-reliance on any single layer. To avoid expensive greedy search, we use a pre-searched configuration to minimize overhead. Experiments show that FAQ consistently outperforms prior methods with negligible extra cost, requiring no backward passes, data reconstruction, or tuning, making it well-suited for edge deployment.

Enhancing Post-Training Quantization via Future Activation Awareness

TL;DR

PTQ for large language models often suffers quantization bias and error accumulation because per-layer scales rely on current activations, a problem amplified when calibration data do not match deployment data. The authors propose Future-Aware Quantization (FAQ), which uses future-layer activations to guide per-layer scales, augmented by a window-wise preview and a pre-searched configuration to balance accuracy and overhead; a theoretical bound suggests reduced quantization error compared with AWQ. Empirically, FAQ yields consistent improvements in perplexity and zero-shot accuracy across diverse models (e.g., Qwen, LLaMA families), with larger gains at ultra-low bit-widths (3-bit) and negligible computational overhead. This approach enables robust, edge-friendly deployment of LLMs by mitigating calibration bias and minimizing the need for backpropagation or data reconstruction during quantization.

Abstract

Post-training quantization (PTQ) is a widely used method to compress large language models (LLMs) without fine-tuning. It typically sets quantization hyperparameters (e.g., scaling factors) based on current-layer activations. Although this method is efficient, it suffers from quantization bias and error accumulation, resulting in suboptimal and unstable quantization, especially when the calibration data is biased. To overcome these issues, we propose Future-Aware Quantization (FAQ), which leverages future-layer activations to guide quantization. This allows better identification and preservation of important weights, while reducing sensitivity to calibration noise. We further introduce a window-wise preview mechanism to softly aggregate multiple future-layer activations, mitigating over-reliance on any single layer. To avoid expensive greedy search, we use a pre-searched configuration to minimize overhead. Experiments show that FAQ consistently outperforms prior methods with negligible extra cost, requiring no backward passes, data reconstruction, or tuning, making it well-suited for edge deployment.
Paper Structure (11 sections, 1 theorem, 9 equations, 1 figure, 3 tables)

This paper contains 11 sections, 1 theorem, 9 equations, 1 figure, 3 tables.

Table of Contents

  1. Introduction
  2. Methodology
  3. Preliminary and Notations
  4. Future-Aware Quantization
  5. Theoretical Foundations of FAQ
  6. Experiments
  7. Experimental Setup
  8. Experimental Results
  9. The impact of calibrate dataset
  10. Conclusion
  11. Acknowledgement

Key Result

Theorem 1

Given the following two assumptions: i) For the average magnitude of activation (per-channel) $\mathbf{a}_i=(a_i^{(1)},...,a_i^{(N)})$ of $N$ channels in the $i$-th layer, the value of the m-th channel is particularly large, that is, $\forall u,u\neq m,a_i^{(m)}\gg a_i^{(u)}$. Meanwhile, for the par where $f_{\mathrm{FAQ}}^1=\mathbf{W}_i\cdot\mathrm{diag}\left(\sum_{l=i}^I\gamma^l\left(\mathbf{a}_

Figures (1)

  • Figure 1: Overview of our proposed method FAQ.

Theorems & Definitions (1)

  • Theorem 1