Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Pack-PTQ: Advancing Post-training Quantization of Neural Networks by Pack-wise Reconstruction

Changjun Li, Runqing Jiang, Zhuo Song, Pengpeng Yu, Ye Zhang, Yulan Guo

TL;DR

Pack-PTQ addresses the key PTQ challenge of losing cross-block dependencies under ultra-low bit-widths by introducing Hessian-guided adaptive packing that forms non-overlapping packs as reconstruction units and a pack-based mixed-precision quantization to tailor bit-widths per pack. The method optimizes a pack-wise reconstruction loss and allocates bits under a memory budget based on pack sensitivity, enabling accurate quantization with limited calibration data. Empirical results on ImageNet and ModelNet40 across CNNs, ViTs, and PointNet show state-of-the-art or competitive accuracy in low-bit settings, with substantial gains over block-wise PTQ methods. This approach improves the practicality of PTQ for diverse vision tasks and edge deployment by preserving accuracy while reducing memory and compute footprints.

Abstract

Post-training quantization (PTQ) has evolved as a prominent solution for compressing complex models, which advocates a small calibration dataset and avoids end-to-end retraining. However, most existing PTQ methods employ block-wise reconstruction, which neglects cross-block dependency and exhibits a notable accuracy drop in low-bit cases. To address these limitations, this paper presents a novel PTQ method, dubbed Pack-PTQ. First, we design a Hessian-guided adaptive packing mechanism to partition blocks into non-overlapping packs, which serve as the base unit for reconstruction, thereby preserving the cross-block dependency and enabling accurate quantization parameters estimation. Second, based on the pack configuration, we propose a mixed-precision quantization approach to assign varied bit-widths to packs according to their distinct sensitivities, thereby further enhancing performance. Extensive experiments on 2D image and 3D point cloud classification tasks, using various network architectures, demonstrate the superiority of our method over the state-of-the-art PTQ methods.

Pack-PTQ: Advancing Post-training Quantization of Neural Networks by Pack-wise Reconstruction

TL;DR

Pack-PTQ addresses the key PTQ challenge of losing cross-block dependencies under ultra-low bit-widths by introducing Hessian-guided adaptive packing that forms non-overlapping packs as reconstruction units and a pack-based mixed-precision quantization to tailor bit-widths per pack. The method optimizes a pack-wise reconstruction loss and allocates bits under a memory budget based on pack sensitivity, enabling accurate quantization with limited calibration data. Empirical results on ImageNet and ModelNet40 across CNNs, ViTs, and PointNet show state-of-the-art or competitive accuracy in low-bit settings, with substantial gains over block-wise PTQ methods. This approach improves the practicality of PTQ for diverse vision tasks and edge deployment by preserving accuracy while reducing memory and compute footprints.

Abstract

Post-training quantization (PTQ) has evolved as a prominent solution for compressing complex models, which advocates a small calibration dataset and avoids end-to-end retraining. However, most existing PTQ methods employ block-wise reconstruction, which neglects cross-block dependency and exhibits a notable accuracy drop in low-bit cases. To address these limitations, this paper presents a novel PTQ method, dubbed Pack-PTQ. First, we design a Hessian-guided adaptive packing mechanism to partition blocks into non-overlapping packs, which serve as the base unit for reconstruction, thereby preserving the cross-block dependency and enabling accurate quantization parameters estimation. Second, based on the pack configuration, we propose a mixed-precision quantization approach to assign varied bit-widths to packs according to their distinct sensitivities, thereby further enhancing performance. Extensive experiments on 2D image and 3D point cloud classification tasks, using various network architectures, demonstrate the superiority of our method over the state-of-the-art PTQ methods.
Paper Structure (21 sections, 1 theorem, 12 equations, 3 figures, 5 tables)

This paper contains 21 sections, 1 theorem, 12 equations, 3 figures, 5 tables.

Table of Contents

  1. Introduction
  2. Related Works
  3. Post-training Quantization
  4. Mixed-precision Quantization
  5. Preliminaries
  6. Quantization
  7. Taylor Expansion
  8. Method
  9. Hessian-guided Adaptive Packing Mechanism
  10. Pack-based Mixed-precision Quantization
  11. Optimization
  12. Experiments
  13. Experimental Setups
  14. Datasets and Competing Methods
  15. Implementation Details
  16. ...and 6 more sections

Key Result

Theorem 1

Assume that $\Delta \mathbf{z}$ is a random vector where each component is i.i.d. as $\mathcal{N}(0, \sigma^2)$. Then the expectation $\mu_{{\mathbf{H}}^{(\mathbf{z})}}$ of all the elements in the Hessian matrix ${\mathbf{H}}^{(\mathbf{z})}$ can be approximated as:

Figures (3)

  • Figure 1: Quantization results of different reconstruction strategies on ImageNet with W3/A3 setting. "No Packing" means employing block-wise reconstruction, "Random Packing" means randomly assigning blocks into packs, and "Fixed-size Packing" means assigning blocks into packs with equal size.
  • Figure 2: Overview of our proposed Pack-PTQ method. We begin by computing individual block scores, which take into account the model loss, block output gradients, and block local loss. Subsequently, we employ our novel packing mechanism to cluster these blocks into packs. Finally, we assign diverse bit-widths to each pack, thereby achieving optimal pack reconstruction and facilitating efficient post-training quantization of neural networks.
  • Figure 3: Visualization of important scores of blocks on (a) ResNet18 and (b) ViT-S. Different colors denote different packs.

Theorems & Definitions (2)

  • Theorem 1
  • proof