Quasar-ViT: Hardware-Oriented Quantization-Aware Architecture Search for Vision Transformers

TL;DR

Quasar-ViT tackles the challenge of deploying accurate Vision Transformers on edge hardware by marrying hardware-aware quantization with neural architecture search. It introduces row-wise flexible mixed-precision quantization, intra-layer weight entanglement, and supernet layer scaling, integrated into a one-shot NAS framework that is guided by FPGA latency and resource models. The approach yields FPGA-validated ViT subnets with high FPS (e.g., up to 251.6 FPS) and competitive ImageNet accuracy, outperforming prior quantization- or NAS-only methods while reducing training cost. This hardware-oriented co-design enables practical ViT deployment on constrained devices and highlights the value of NAS-guided quantization for transformer-based vision workstreams.

Abstract

Vision transformers (ViTs) have demonstrated their superior accuracy for computer vision tasks compared to convolutional neural networks (CNNs). However, ViT models are often computation-intensive for efficient deployment on resource-limited edge devices. This work proposes Quasar-ViT, a hardware-oriented quantization-aware architecture search framework for ViTs, to design efficient ViT models for hardware implementation while preserving the accuracy. First, Quasar-ViT trains a supernet using our row-wise flexible mixed-precision quantization scheme, mixed-precision weight entanglement, and supernet layer scaling techniques. Then, it applies an efficient hardware-oriented search algorithm, integrated with hardware latency and resource modeling, to determine a series of optimal subnets from supernet under different inference latency targets. Finally, we propose a series of model-adaptive designs on the FPGA platform to support the architecture search and mitigate the gap between the theoretical computation reduction and the practical inference speedup. Our searched models achieve 101.5, 159.6, and 251.6 frames-per-second (FPS) inference speed on the AMD/Xilinx ZCU102 FPGA with 80.4%, 78.6%, and 74.9% top-1 accuracy, respectively, for the ImageNet dataset, consistently outperforming prior works.

Figures (9)

• Figure 1: Transformer encoder block structure and ViT model execution performance profiling on CPU for a) DeiT-Base and b) DeiT-S with 12 encoders on ImageNet dataset.
• Figure 2: Comparison of mixed-precision quantization under different granularity. We use the example of two different bit-widths, represented as blue and pink colors. We propose the row-wise flexible mixed-precision quantization in (d).
• Figure 3: Classic weight sharing and original weight entanglement versus our proposed mixed-precision weight entanglement.
• Figure 4: Supernet layer scaling (SLS) in Quasar-ViT encoder block. We use the SLS after MLP as an example.
• Figure 5: SuperNet training process including candidate sampling and the searched results for different targeting FPS. We use a model with layers of 6 channels as a toy example.