APHQ-ViT: Post-Training Quantization with Average Perturbation Hessian Based Reconstruction for Vision Transformers

TL;DR

This work tackles the accuracy degradation of Vision Transformers under post-training quantization by revealing two core bottlenecks: misestimated output importance in reconstruction-based PTQ and severe quantization errors from post-GELU activations. It introduces APHQ-ViT, which combines Average Perturbation Hessian (APH) loss for stable, task-agnostic output sensitivity estimation with MLP Reconstruction (MR) that substitutes GELU with ReLU and reconstructs MLP outputs using APH-guided losses and percentile-based clamping. The method uses only unlabeled calibration data and uniform linear quantizers, achieving substantial improvements at 3-bit and 4-bit across ViT, DeiT, and Swin architectures on ImageNet and COCO, often outperforming state-of-the-art methods. The approach offers practical deployment benefits through improved accuracy, reduced activation ranges, and maintained efficiency, with MR also contributing to faster inference when deployed on hardware supporting uniform quantization.

Abstract

Vision Transformers (ViTs) have become one of the most commonly used backbones for vision tasks. Despite their remarkable performance, they often suffer significant accuracy drops when quantized for practical deployment, particularly by post-training quantization (PTQ) under ultra-low bits. Recently, reconstruction-based PTQ methods have shown promising performance in quantizing Convolutional Neural Networks (CNNs). However, they fail when applied to ViTs, primarily due to the inaccurate estimation of output importance and the substantial accuracy degradation in quantizing post-GELU activations. To address these issues, we propose \textbf{APHQ-ViT}, a novel PTQ approach based on importance estimation with Average Perturbation Hessian (APH). Specifically, we first thoroughly analyze the current approximation approaches with Hessian loss, and propose an improved average perturbation Hessian loss. To deal with the quantization of the post-GELU activations, we design an MLP Reconstruction (MR) method by replacing the GELU function in MLP with ReLU and reconstructing it by the APH loss on a small unlabeled calibration set. Extensive experiments demonstrate that APHQ-ViT using linear quantizers outperforms existing PTQ methods by substantial margins in 3-bit and 4-bit across different vision tasks. The source code is available at https://github.com/GoatWu/APHQ-ViT.

Figures (5)

• Figure 1: (a) shows the box plot of activations for each linear layer in vit-small.blocks.5 and 6 by using the 0.99 quantile, highlighting the varying ranges of post-GELU activations. (b) and (c) display that the activation range of fc2 is significantly reduced after MLP Reconstruction, while the weight range only exhibits slight changes.
• Figure 2: Framework overview of APHQ-ViT. In the block-wise quantization process, we first reconstruct the MLP layer, followed by quantization reconstruction, both of which are optimized by the proposed Average Perturbation Hessian (APH) loss. The MLP Reconstruction (MR) method replaces the GELU activation function with ReLU and reduces the post-GELU activation range. The detailed implementation of the APH loss is visualized at the bottom.
• Figure A: The loss curve of ViT-Small-blocks.6 on W3/A3.
• Figure B: Illustration on the token importance in ViT-S.blocks.7.
• Figure C: Illustration on the channel importance.

Theorems & Definitions (4)

• Theorem 3.1
• Theorem 3.2
• proof
• proof