ADFQ-ViT: Activation-Distribution-Friendly Post-Training Quantization for Vision Transformers

TL;DR

This paper tackles the problem of aggressive accuracy loss when quantizing Vision Transformers to low bit-width by analyzing distinctive activation distributions. It introduces ADFQ-ViT, a framework that fuses three components—Per-Patch Outlier-aware Quantizer, Shift-Log2 Quantizer, and Attention-score enhanced Module-wise Optimization—to tailor quantization to ViT activation characteristics. The approach yields consistent 4-bit gains over strong baselines across image classification, object detection, instance segmentation, and zero-shot segmentation with SAM, and achieves 6-bit performance close to full precision. The work enables practical deployment of ViTs on resource-limited devices by significantly improving low-bit quantization viability with minimal architectural changes.

Abstract

Vision Transformers (ViTs) have exhibited exceptional performance across diverse computer vision tasks, while their substantial parameter size incurs significantly increased memory and computational demands, impeding effective inference on resource-constrained devices. Quantization has emerged as a promising solution to mitigate these challenges, yet existing methods still suffer from significant accuracy loss at low-bit. We attribute this issue to the distinctive distributions of post-LayerNorm and post-GELU activations within ViTs, rendering conventional hardware-friendly quantizers ineffective, particularly in low-bit scenarios. To address this issue, we propose a novel framework called Activation-Distribution-Friendly post-training Quantization for Vision Transformers, ADFQ-ViT. Concretely, we introduce the Per-Patch Outlier-aware Quantizer to tackle irregular outliers in post-LayerNorm activations. This quantizer refines the granularity of the uniform quantizer to a per-patch level while retaining a minimal subset of values exceeding a threshold at full-precision. To handle the non-uniform distributions of post-GELU activations between positive and negative regions, we design the Shift-Log2 Quantizer, which shifts all elements to the positive region and then applies log2 quantization. Moreover, we present the Attention-score enhanced Module-wise Optimization which adjusts the parameters of each quantizer by reconstructing errors to further mitigate quantization error. Extensive experiments demonstrate ADFQ-ViT provides significant improvements over various baselines in image classification, object detection, and instance segmentation tasks at 4-bit. Specifically, when quantizing the ViT-B model to 4-bit, we achieve a 10.23% improvement in Top-1 accuracy on the ImageNet dataset.

Figures (9)

• Figure 1: Overview of our ADFQ-ViT framework. We apply Per-Patch Outlier-aware Quantizer for post-LayerNorm activations (QKV and FC1 Linear), Shift-Log2 Quantizer for post-GELU activations (FC2 Linear), and Attention-score enhanced Module-wise Optimization for optimizing the parameters of weight and input activation quantizers.
• Figure 2: Visualization of the absolute value distribution for post-LayerNorm activations. We choose the DeiT-S model and select 32 images from ImageNet dataset for inference to visualize absolute value of the post-LayerNorm activations in its 1st (first) and 12th (last) blocks.
• Figure 3: Comparison of different quantization granularity for post-LayerNorm activations $X_{ln}$. The red dashed boxes represent the statistical ranges of the quantization parameters.
• Figure 4: Ratio of outliers for input activations (post-LayerNorm activations) of QKV Linear and FC1 Liner for the Deit-S model.
• Figure 5: Visualization of the value distribution for post-GELU activations. The red dotted line indicates the dividing line between positive and negative values.