Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LoaQ: Layer-wise Output Approximation Quantization

Li Lin, Xiaojun Wan

TL;DR

LoaQ addresses misalignment between traditional layer-wise PTQ objectives (weight or linear-output focus) and the full-model outputs in large language models. It introduces output-matching factors at linear-layer and sub-block levels, incorporating RMSNorm awareness to align the quantized outputs with the originals and deriving a simple, closed-form update compatible with existing pipelines. The method is validated on LLaMA and Qwen families, showing robust gains for both weight-only and weight-activation quantization, and benefits from integration with Hadamard transforms and NeUQI initialization. Overall, LoaQ demonstrates consistent improvements in quantization quality across tasks and settings, signaling a meaningful advance in layer-wise post-training quantization for large-scale models.

Abstract

A natural and intuitive idea in model quantization is to approximate each component's quantized output to match its original. Motivated by this idea, most layer-wise post-training quantization (PTQ) methods focus on weight approximation at the linear-layer level. As a result, this local objective often yields insufficient approximations and practical deviations from the guiding intuition. Recent work has improved the approximation of linear-layer outputs within the layer-wise PTQ framework, but such refinements remain inadequate for achieving alignment with the full-model output. Based on a deeper understanding of the structure of mainstream LLMs, we propose LoaQ, which incorporates output-matching factors when quantizing linear layers within the layer-wise PTQ framework. It better aligns with this intuition and can feature a simple closed-form solution, making it orthogonal to existing techniques and readily integrable into existing quantization pipelines. Experiments on the LLaMA and Qwen model families demonstrate that LoaQ performs effectively in both weight-only and weight-activation quantization. By integrating seamlessly with existing quantization strategies, it further enhances overall quantization quality and shows strong potential to advance the frontier of post-training quantization.

LoaQ: Layer-wise Output Approximation Quantization

TL;DR

LoaQ addresses misalignment between traditional layer-wise PTQ objectives (weight or linear-output focus) and the full-model outputs in large language models. It introduces output-matching factors at linear-layer and sub-block levels, incorporating RMSNorm awareness to align the quantized outputs with the originals and deriving a simple, closed-form update compatible with existing pipelines. The method is validated on LLaMA and Qwen families, showing robust gains for both weight-only and weight-activation quantization, and benefits from integration with Hadamard transforms and NeUQI initialization. Overall, LoaQ demonstrates consistent improvements in quantization quality across tasks and settings, signaling a meaningful advance in layer-wise post-training quantization for large-scale models.

Abstract

A natural and intuitive idea in model quantization is to approximate each component's quantized output to match its original. Motivated by this idea, most layer-wise post-training quantization (PTQ) methods focus on weight approximation at the linear-layer level. As a result, this local objective often yields insufficient approximations and practical deviations from the guiding intuition. Recent work has improved the approximation of linear-layer outputs within the layer-wise PTQ framework, but such refinements remain inadequate for achieving alignment with the full-model output. Based on a deeper understanding of the structure of mainstream LLMs, we propose LoaQ, which incorporates output-matching factors when quantizing linear layers within the layer-wise PTQ framework. It better aligns with this intuition and can feature a simple closed-form solution, making it orthogonal to existing techniques and readily integrable into existing quantization pipelines. Experiments on the LLaMA and Qwen model families demonstrate that LoaQ performs effectively in both weight-only and weight-activation quantization. By integrating seamlessly with existing quantization strategies, it further enhances overall quantization quality and shows strong potential to advance the frontier of post-training quantization.

Paper Structure

This paper contains 33 sections, 15 equations, 4 figures, 7 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Layer-wise PTQ
  4. Error-compensated PTQ
  5. Preliminaries
  6. Linear-layer Loss
  7. GPTQ
  8. GPTAQ
  9. Sub-block
  10. Methodology
  11. Linear Layer Output Approximation
  12. Sub-block Output Approximation
  13. Normalized Sub-block Output Approximation
  14. Tuning Parameters
  15. Experiment
  16. ...and 18 more sections

Figures (4)

  • Figure 1: The overall diagram of LoaQ. Each colored vertical bidirectional arrow denotes the corresponding approximate output obtained when quantizing the weight matrix of the same color.
  • Figure 2: Results under 3-bit quantization on LLaMA 3.2 1B, showing the mean squared error (MSE) of sub-blocks' outputs. The left plot reports the MSE of unnormalized outputs, while the right plot reports the MSE of normalized outputs.
  • Figure 3: Results under 3-bit quantization on LLaMA 3.2 1B, reporting average accuracy across different values of $\alpha$ and $\beta$.
  • Figure 4: Results of 2-bit channel-wise quantization on LLaMA 2 7B, presenting the mean squared error (MSE) of Transformer block outputs excluding outlier tokens. The top two plots show the MSE without the Hadamard transform, while the bottom two plots show the MSE with the Hadamard transform. For each row, the left plot corresponds to unnormalized outputs, and the right plot corresponds to normalized outputs.