BHViT: Binarized Hybrid Vision Transformer

TL;DR

BHViT proposes a binarization-friendly hybrid Vision Transformer that overcomes the performance gap of binary ViTs by integrating three targeted innovations: MSGDC for multi-scale token mixing in early stages, MSMHA for efficient global-local attention in later stages, and a shift-based Binary MLP enhancement. A Quantization Decomposition scheme augments binarized attention by producing multiple binary attention maps, while a layer-wise residual structure and a dedicated regularization loss mitigate optimization challenges in binary training. The approach yields state-of-the-art results among binary ViTs on ImageNet-1K and achieves superior segmentation performance on RS-LVF and ADE20K, with competitive edge deployment metrics. The work demonstrates that a carefully designed hybrid architecture, coupled with principled binarization techniques, can deliver high accuracy with low computational demand suitable for edge devices.

Abstract

Model binarization has made significant progress in enabling real-time and energy-efficient computation for convolutional neural networks (CNN), offering a potential solution to the deployment challenges faced by Vision Transformers (ViTs) on edge devices. However, due to the structural differences between CNN and Transformer architectures, simply applying binary CNN strategies to the ViT models will lead to a significant performance drop. To tackle this challenge, we propose BHViT, a binarization-friendly hybrid ViT architecture and its full binarization model with the guidance of three important observations. Initially, BHViT utilizes the local information interaction and hierarchical feature aggregation technique from coarse to fine levels to address redundant computations stemming from excessive tokens. Then, a novel module based on shift operations is proposed to enhance the performance of the binary Multilayer Perceptron (MLP) module without significantly increasing computational overhead. In addition, an innovative attention matrix binarization method based on quantization decomposition is proposed to evaluate the token's importance in the binarized attention matrix. Finally, we propose a regularization loss to address the inadequate optimization caused by the incompatibility between the weight oscillation in the binary layers and the Adam Optimizer. Extensive experimental results demonstrate that our proposed algorithm achieves SOTA performance among binary ViT methods.

Figures (14)

• Figure 1: The classification accuracy on ImageNet-1K dataset of the binary method, ReActNet liu2020reactnet, when applied in CNN and ViT architectures, respectively.
• Figure 1: The Schematic diagram of the process of computing $x_i$, referring to Eq. \ref{['pdf']}.
• Figure 2: The multiplication between binary vectors can be implemented by the Xnor and popcount. The result is $2p-n$
• Figure 2: The Schematic diagram of DeMoivre–Laplace theorem with different $d$. The histograms display the Binomial distributions with the same $p=0.5$ and different $d$. The red lines are the corresponding fitted Gaussian distributions. When $d$ increases, the Binomial distribution can be better approximated by the Gaussian distribution.
• Figure 3: The architecture of the proposed binary hybrid ViT. "MSGDC" and "MSMHA" refer to the Binary Multi-Scale Grouped Dilated Convolution module and the Binary Multi-Scale Multi-Head Attention module, respectively. "GAP" stands for global average pooling. "Input_D" denotes the input tensor after downsampling. "RPRELU" is the activation layer proposed in liu2020reactnet.