Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

LPCD: Unified Framework from Layer-Wise to Submodule Quantization

Yuma Ichikawa, Yudai Fujimoto, Akira Sakai

TL;DR

LPCD introduces a unified submodule-centric quantization framework that extends post-training quantization beyond layers by optimizing submodules in the output space and projecting back with existing layer-wise quantizers. It unifies QEP, LoaQ, and classical layer-wise PTQ as special cases, and demonstrates substantial reductions in quantization error across QK, VO, and Up-Down submodules in large Transformer models. Experiments on LLaMA and Qwen show LPCD consistently improves perplexity and zero-shot accuracy across bit-widths, particularly at 3-2 bits, while preserving compatibility with standard PTQ pipelines. The approach offers a scalable, hardware-friendly path to deploy ultra-low-bit LLMs and suggests future extensions to nonlinear submodules and KV-cache joint quantization.

Abstract

Post-training quantization (PTQ) aims to preserve model-level behavior; however, most methods focus on individual linear layers. Even recent extensions, such as QEP and LoaQ, which mitigate error propagation or target specific submodules, still rely on layer-wise formulations and fail to capture the behavior of larger submodules. We introduce Layer-Projected Coordinate Descent (LPCD), a unified framework that extends PTQ beyond layers by optimizing relaxed objectives across arbitrary submodules and projecting the solutions with layer-wise quantizers. LPCD generalizes existing methods and provides a principled approach to quantizing complex submodules while maintaining the efficiency and compatibility of layer-wise PTQ pipelines. Across diverse LLM architectures and bit-widths, LPCD-based submodule quantization consistently enhances both layer-wise PTQ methods and existing submodule approaches.

LPCD: Unified Framework from Layer-Wise to Submodule Quantization

TL;DR

LPCD introduces a unified submodule-centric quantization framework that extends post-training quantization beyond layers by optimizing submodules in the output space and projecting back with existing layer-wise quantizers. It unifies QEP, LoaQ, and classical layer-wise PTQ as special cases, and demonstrates substantial reductions in quantization error across QK, VO, and Up-Down submodules in large Transformer models. Experiments on LLaMA and Qwen show LPCD consistently improves perplexity and zero-shot accuracy across bit-widths, particularly at 3-2 bits, while preserving compatibility with standard PTQ pipelines. The approach offers a scalable, hardware-friendly path to deploy ultra-low-bit LLMs and suggests future extensions to nonlinear submodules and KV-cache joint quantization.

Abstract

Post-training quantization (PTQ) aims to preserve model-level behavior; however, most methods focus on individual linear layers. Even recent extensions, such as QEP and LoaQ, which mitigate error propagation or target specific submodules, still rely on layer-wise formulations and fail to capture the behavior of larger submodules. We introduce Layer-Projected Coordinate Descent (LPCD), a unified framework that extends PTQ beyond layers by optimizing relaxed objectives across arbitrary submodules and projecting the solutions with layer-wise quantizers. LPCD generalizes existing methods and provides a principled approach to quantizing complex submodules while maintaining the efficiency and compatibility of layer-wise PTQ pipelines. Across diverse LLM architectures and bit-widths, LPCD-based submodule quantization consistently enhances both layer-wise PTQ methods and existing submodule approaches.

Paper Structure

This paper contains 66 sections, 1 theorem, 106 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Notation
  5. Layer-wise PTQ
  6. Direct Weight Quantization.
  7. Activation-Aware Quantization.
  8. Quantization Error Propagation
  9. Residual Path Quantization
  10. Method
  11. Problem Setting
  12. Layer-Projected Coordinate Descent
  13. Relaxation Step.
  14. Projection Step.
  15. QEP as a Single LPCD Update
  16. ...and 51 more sections

Key Result

proposition 1

Consider the objective defined in Eq. eq:QEP-LPCD-objective with blocks $M_{1}=\widehat{W}$ and $M_{2}=\widehat{X}$. Fix the activation block $\widehat{X}$ and perform a single LPCD update on the weight block $\widehat{W}$. Let $\widehat{W}^{(1)}$ denote the value of the weight block following this

Figures (2)

  • Figure 1: Output MSE across Transformer blocks in Llama 3 8B with 4, 3, and 2 bit weight quantization. LPCD consistently yields lower quantization error than QEP and LoaQ.
  • Figure 2: Conceptual diagram of the submodules considered in this work using LPCD; the regions enclosed by the red dashed boxes correspond to submodules.

Theorems & Definitions (3)

  • proposition 1
  • proof
  • remark 1