Trio-ViT: Post-Training Quantization and Acceleration for Softmax-Free Efficient Vision Transformer

TL;DR

Trio-ViT targets the quantization and acceleration of Softmax-free efficient Vision Transformers (EfficientViT). It introduces a tailored post-training quantization engine with channel-wise migration, filter-wise shifting, and log2 quantization, paired with a hybrid MAT+R-MAC accelerator and inter-/intra-layer pipelines. Empirical results on ImageNet show only about a $0.92\%$ accuracy drop with 8-bit quantization, while delivering up to $3.6\times$, $5.0\times$, and $7.3\times$ FPS and notable DSP/energy efficiency gains over SOTA baselines; Cityscapes semantic segmentation results also remain robust. The work demonstrates a practical path to deploying Softmax-free EfficientViT models on edge hardware, and provides public code for replication and extension.

Abstract

Motivated by the huge success of Transformers in the field of natural language processing (NLP), Vision Transformers (ViTs) have been rapidly developed and achieved remarkable performance in various computer vision tasks. However, their huge model sizes and intensive computations hinder ViTs' deployment on embedded devices, calling for effective model compression methods, such as quantization. Unfortunately, due to the existence of hardware-unfriendly and quantization-sensitive non-linear operations, particularly {Softmax}, it is non-trivial to completely quantize all operations in ViTs, yielding either significant accuracy drops or non-negligible hardware costs. In response to challenges associated with \textit{standard ViTs}, we focus our attention towards the quantization and acceleration for \textit{efficient ViTs}, which not only eliminate the troublesome Softmax but also integrate linear attention with low computational complexity, and propose Trio-ViT accordingly. Specifically, at the algorithm level, we develop a {tailored post-training quantization engine} taking the unique activation distributions of Softmax-free efficient ViTs into full consideration, aiming to boost quantization accuracy. Furthermore, at the hardware level, we build an accelerator dedicated to the specific Convolution-Transformer hybrid architecture of efficient ViTs, thereby enhancing hardware efficiency. Extensive experimental results consistently prove the effectiveness of our Trio-ViT framework. {Particularly, we can gain up to $\uparrow$$\mathbf{3.6}\times$, $\uparrow$$\mathbf{5.0}\times$, and $\uparrow$$\mathbf{7.3}\times$ FPS under comparable accuracy over state-of-the-art ViT accelerators, as well as $\uparrow$$\mathbf{6.0}\times$, $\uparrow$$\mathbf{1.5}\times$, and $\uparrow$$\mathbf{2.1}\times$ DSP efficiency.} Codes are available at \url{https://github.com/shihuihong214/Trio-ViT}.

Figures (10)

• Figure 1: The architecture of standard ViTs vitdeit that include multiple Transformer blocks. Each block includes an MHSA and an MLP. 'MatMul.' is the abbreviation for 'Matrix Multiplication'.
• Figure 2: The architecture of EfficientViT cai2022efficientvit that mainly compromises MBConvs Sandler2018MobileNetV2IR and EfficientViT modules.
• Figure 3: The minimum/maximum values along the channel dimension as well as distributions of input activations in (a) (b) the first PWConv (PW1), (c) (d) DWConv (DW), and (e) (f) the second PWConv (PW2), respectively, within the MBConv in the last stage of EfficientViT-B1 cai2022efficientvit.
• Figure 4: (a) gives the distribution of divisor within the MSA from the last stage of EfficientViT-B1 cai2022efficientvit, with visualizations of quantization bins of uniform and log2 quantization. (b) shows quantization bins of small values near zero.
• Figure 5: Illustrating the layer-wise quantization for activations and the filter-wise quantization for (a) generic convolution (Conv), (b) DWConv, and (c) PWConv. Pixels represented in different colors undergo quantization with distinct scaling factors.