Progressive Fine-to-Coarse Reconstruction for Accurate Low-Bit Post-Training Quantization in Vision Transformers

TL;DR

This work tackles the accuracy gap faced by Vision Transformers under low-bit post-training quantization (PTQ). It introduces Progressive Fine-to-Coarse Reconstruction (PFCR), which reconstructs ViT components from fine granularity (MHSA and MLP with shortcuts) to coarser blocks in an iterative, progressive manner, thereby reducing reconstruction error. To further enhance performance, the authors propose a Progressive Optimization Strategy (POS) that combines a two-stage training regime with a diminishing granularity, improving stability and convergence. Empirical results on ImageNet and COCO show state-of-the-art Top-1 accuracy for 3-bit and 4-bit quantization across ViT, DeiT, and Swin backbones, with notable gains over prior methods and demonstrated generalization to high-level vision tasks. The approach offers a practical path to highly efficient ViTs for edge devices, though it incurs additional memory costs and may require careful hyper-parameter tuning for very large models or different hardware constraints.

Abstract

Due to its efficiency, Post-Training Quantization (PTQ) has been widely adopted for compressing Vision Transformers (ViTs). However, when quantized into low-bit representations, there is often a significant performance drop compared to their full-precision counterparts. To address this issue, reconstruction methods have been incorporated into the PTQ framework to improve performance in low-bit quantization settings. Nevertheless, existing related methods predefine the reconstruction granularity and seldom explore the progressive relationships between different reconstruction granularities, which leads to sub-optimal quantization results in ViTs. To this end, in this paper, we propose a Progressive Fine-to-Coarse Reconstruction (PFCR) method for accurate PTQ, which significantly improves the performance of low-bit quantized vision transformers. Specifically, we define multi-head self-attention and multi-layer perceptron modules along with their shortcuts as the finest reconstruction units. After reconstructing these two fine-grained units, we combine them to form coarser blocks and reconstruct them at a coarser granularity level. We iteratively perform this combination and reconstruction process, achieving progressive fine-to-coarse reconstruction. Additionally, we introduce a Progressive Optimization Strategy (POS) for PFCR to alleviate the difficulty of training, thereby further enhancing model performance. Experimental results on the ImageNet dataset demonstrate that our proposed method achieves the best Top-1 accuracy among state-of-the-art methods, particularly attaining 75.61% for 3-bit quantized ViT-B in PTQ. Besides, quantization results on the COCO dataset reveal the effectiveness and generalization of our proposed method on other computer vision tasks like object detection and instance segmentation.

Figures (5)

• Figure 1: Comparisons between the conventional reconstruction methods with the single and fixed granularity and our proposed progressive fine-to-coarse reconstruction method.
• Figure 2: Illustration of the proposed PFCR. The left sub-figure shows the two finest reconstruction units with granularity level $g=0$. The right sub-figure demonstrates the iterative process of reconstruction and combination from fine granularity to coarse granularity progressively.
• Figure 3: Comparison of the reconstruction loss distribution in the 12-block 4-bit quantized ViT-S between conventional block-wise reconstruction and the proposed PFCR. (a) The overall loss distribution of 12 blocks in ViT-S. (b) The zoomed in loss distribution of the first 6 blocks in ViT-S.
• Figure 4: Quantization results of ViT-S in 3-bit, 4-bit, and 6-bit bit-width with increasing iteration number. (a) The Top-1 (%) accuracy and (b) the time cost (minutes) is adopted as the evaluation metric for performance and efficiency respectively. We test three times to compute the mean and std.
• Figure 5: Qualitative visualization results of 3-bit quantized ViT-B using I&S-ViT method and our proposed PFCR. We select 8 images from the validation set of ImageNet dataset and utilize the Grad-CAM GradCamselvaraju2017grad technique to derive the attention maps.