Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

EPTQ: Enhanced Post-Training Quantization via Hessian-guided Network-wise Optimization

Ofir Gordon, Elad Cohen, Hai Victor Habi, Arnon Netzer

TL;DR

This work tackles the challenge of deploying DNNs on edge devices by enhancing Post-Training Quantization (PTQ) through network-wide optimization that accounts for cross-layer dependencies. It introduces Enhanced PTQ ($EPTQ$), which leverages a label-free Hessian bound to construct a Sample-Layer Attention ($SLA$) score and a Hessian-based MSE ($Hmse$) metric to guide weight quantization and the rounding policy. The method combines a Hessian-guided knowledge-distillation loss with a gradual activation quantization scheme to recover accuracy lost during quantization, achieving state-of-the-art performance on ImageNet, COCO, and Pascal VOC without data augmentation. The approach is validated across diverse architectures and tasks, with ablations showing the pivotal roles of SLA, Hmse, and the gradual quantization strategy, and the authors provide public code for reproducibility.

Abstract

Quantization is a key method for deploying deep neural networks on edge devices with limited memory and computation resources. Recent improvements in Post-Training Quantization (PTQ) methods were achieved by an additional local optimization process for learning the weight quantization rounding policy. However, a gap exists when employing network-wise optimization with small representative datasets. In this paper, we propose a new method for enhanced PTQ (EPTQ) that employs a network-wise quantization optimization process, which benefits from considering cross-layer dependencies during optimization. EPTQ enables network-wise optimization with a small representative dataset using a novel sample-layer attention score based on a label-free Hessian matrix upper bound. The label-free approach makes our method suitable for the PTQ scheme. We give a theoretical analysis for the said bound and use it to construct a knowledge distillation loss that guides the optimization to focus on the more sensitive layers and samples. In addition, we leverage the Hessian upper bound to improve the weight quantization parameters selection by focusing on the more sensitive elements in the weight tensors. Empirically, by employing EPTQ we achieve state-of-the-art results on various models, tasks, and datasets, including ImageNet classification, COCO object detection, and Pascal-VOC for semantic segmentation.

EPTQ: Enhanced Post-Training Quantization via Hessian-guided Network-wise Optimization

TL;DR

This work tackles the challenge of deploying DNNs on edge devices by enhancing Post-Training Quantization (PTQ) through network-wide optimization that accounts for cross-layer dependencies. It introduces Enhanced PTQ (), which leverages a label-free Hessian bound to construct a Sample-Layer Attention () score and a Hessian-based MSE () metric to guide weight quantization and the rounding policy. The method combines a Hessian-guided knowledge-distillation loss with a gradual activation quantization scheme to recover accuracy lost during quantization, achieving state-of-the-art performance on ImageNet, COCO, and Pascal VOC without data augmentation. The approach is validated across diverse architectures and tasks, with ablations showing the pivotal roles of SLA, Hmse, and the gradual quantization strategy, and the authors provide public code for reproducibility.

Abstract

Quantization is a key method for deploying deep neural networks on edge devices with limited memory and computation resources. Recent improvements in Post-Training Quantization (PTQ) methods were achieved by an additional local optimization process for learning the weight quantization rounding policy. However, a gap exists when employing network-wise optimization with small representative datasets. In this paper, we propose a new method for enhanced PTQ (EPTQ) that employs a network-wise quantization optimization process, which benefits from considering cross-layer dependencies during optimization. EPTQ enables network-wise optimization with a small representative dataset using a novel sample-layer attention score based on a label-free Hessian matrix upper bound. The label-free approach makes our method suitable for the PTQ scheme. We give a theoretical analysis for the said bound and use it to construct a knowledge distillation loss that guides the optimization to focus on the more sensitive layers and samples. In addition, we leverage the Hessian upper bound to improve the weight quantization parameters selection by focusing on the more sensitive elements in the weight tensors. Empirically, by employing EPTQ we achieve state-of-the-art results on various models, tasks, and datasets, including ImageNet classification, COCO object detection, and Pascal-VOC for semantic segmentation.
Paper Structure (44 sections, 1 theorem, 22 equations, 5 figures, 8 tables, 3 algorithms)

This paper contains 44 sections, 1 theorem, 22 equations, 5 figures, 8 tables, 3 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Post-Training Quantization Optimization
  4. Hessian for Model Compression
  5. Preliminaries
  6. Notations.
  7. Network and Loss function.
  8. Hessian matrix approximation.
  9. Method
  10. Sample-Layer Attention Bound
  11. Hessian-based Mean Squared Error
  12. Enhanced PTQ (EPTQ) Process
  13. Threshold Selection.
  14. Hessian-aware Network-wise optimization loss.
  15. Gradual Activation Quantization.
  16. ...and 29 more sections

Key Result

Proposition 1

Assume that a network $f$ minimizes task loss $\mathcal{L}_{task}$ such that the following assumption: ${\left.\nabla_{\bm{r}}\mathcal{L}_{task}\left(\bm{y},\bm{r} \right)\right\vert_{\bm{r}=f\left(\bm{x}\right)} \approx 0}$ is satisfied. In addition, given that Assumption asm:lfh holds, we get that and if there exists a constant $c>0$ such that $\mathbf{A}\left(r\right)\preceq c\mathbf{I}$, then:

Figures (5)

  • Figure 1: Sample-Layer Attention illustration for network-wise optimization, on ResNet18 with 16 random samples. (1) On the left, there is a demonstration of the values of the computed Hessian bound scores w.r.t. layers outputs. The zoomed area shows how various samples yield scores of varying magnitudes across different layers. (2) The top-right part presents the scores for different samples for a given layer, showcasing the per-sample attention. (3) The bottom-right part presents the scores for all layers for a given sample, showcasing the per-layer attention. Note that all values are in log-scale.
  • Figure 2: Relative accuracy drop of several classification networks quantized with 3 bits for weights and activation with a different number of images.
  • Figure 3: Relative accuracy drop of several classification networks quantized with 3 bits for weights and activation with a different number of iterations.
  • Figure 4: Comparison between Hessian trace approximation following the LFH assumption, and the true Hessian trace yao2020pyhessian of different loss functions on ResNet18 on random ImageNet samples. In (a) the values of the Hessian trace are normalized, while in (b) the values are normalized using log normalization.)
  • Figure 5: A comparison between the Hessian trace of common task loss functions and our task-invariant Label-Free Hessian approximation on MobileNet-V2 and ResNet50, computed on a random set from ImageNet.

Theorems & Definitions (2)

  • Proposition 1: Label-Free Hessian
  • proof