Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An FPGA-Based Reconfigurable Accelerator for Convolution-Transformer Hybrid EfficientViT

Haikuo Shao, Huihong Shi, Wendong Mao, Zhongfeng Wang

TL;DR

This work tackles the challenge of accelerating EfficientViT, a Convolution-Transformer hybrid ViT, on resource-limited devices by designing a reconfigurable FPGA accelerator. The architecture combines a Reconfigurable Processing Element and a MAT engine with a time-multiplexed, pipelined dataflow to fuse computations across convolution blocks and the lightweight MSA. It supports DWConvs, PWConvs, and MatMuls, and introduces intra- and inter-layer fusion to reduce off-chip data transfers. Implemented on a Xilinx ZCU102 at $200$MHz, it achieves up to $780.2$ GOPS throughput and $105.1$ GOPS/W energy efficiency, outperforming CPU- and other accelerator-based baselines and advancing practical EfficientViT deployment.

Abstract

Vision Transformers (ViTs) have achieved significant success in computer vision. However, their intensive computations and massive memory footprint challenge ViTs' deployment on embedded devices, calling for efficient ViTs. Among them, EfficientViT, the state-of-the-art one, features a Convolution-Transformer hybrid architecture, enhancing both accuracy and hardware efficiency. Unfortunately, existing accelerators cannot fully exploit the hardware benefits of EfficientViT due to its unique architecture. In this paper, we propose an FPGA-based accelerator for EfficientViT to advance the hardware efficiency frontier of ViTs. Specifically, we design a reconfigurable architecture to efficiently support various operation types, including lightweight convolutions and attention, boosting hardware utilization. Additionally, we present a time-multiplexed and pipelined dataflow to facilitate both intra- and inter-layer fusions, reducing off-chip data access costs. Experimental results show that our accelerator achieves up to 780.2 GOPS in throughput and 105.1 GOPS/W in energy efficiency at 200MHz on the Xilinx ZCU102 FPGA, which significantly outperforms prior works.

An FPGA-Based Reconfigurable Accelerator for Convolution-Transformer Hybrid EfficientViT

TL;DR

This work tackles the challenge of accelerating EfficientViT, a Convolution-Transformer hybrid ViT, on resource-limited devices by designing a reconfigurable FPGA accelerator. The architecture combines a Reconfigurable Processing Element and a MAT engine with a time-multiplexed, pipelined dataflow to fuse computations across convolution blocks and the lightweight MSA. It supports DWConvs, PWConvs, and MatMuls, and introduces intra- and inter-layer fusion to reduce off-chip data transfers. Implemented on a Xilinx ZCU102 at MHz, it achieves up to GOPS throughput and GOPS/W energy efficiency, outperforming CPU- and other accelerator-based baselines and advancing practical EfficientViT deployment.

Abstract

Vision Transformers (ViTs) have achieved significant success in computer vision. However, their intensive computations and massive memory footprint challenge ViTs' deployment on embedded devices, calling for efficient ViTs. Among them, EfficientViT, the state-of-the-art one, features a Convolution-Transformer hybrid architecture, enhancing both accuracy and hardware efficiency. Unfortunately, existing accelerators cannot fully exploit the hardware benefits of EfficientViT due to its unique architecture. In this paper, we propose an FPGA-based accelerator for EfficientViT to advance the hardware efficiency frontier of ViTs. Specifically, we design a reconfigurable architecture to efficiently support various operation types, including lightweight convolutions and attention, boosting hardware utilization. Additionally, we present a time-multiplexed and pipelined dataflow to facilitate both intra- and inter-layer fusions, reducing off-chip data access costs. Experimental results show that our accelerator achieves up to 780.2 GOPS in throughput and 105.1 GOPS/W in energy efficiency at 200MHz on the Xilinx ZCU102 FPGA, which significantly outperforms prior works.
Paper Structure (14 sections, 6 figures, 2 tables)

This paper contains 14 sections, 6 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Structure of EfficientViT
  3. Proposed Hardware Design
  4. Multipliers and Adder-Trees Design Paradigm
  5. Reconfigurable Architecture Design
  6. DW Mode
  7. PW Mode
  8. Overall Architecture of Proposed Accelerator
  9. Time-Multiplexed and Pipelined Dataflow
  10. Experimental Results
  11. Experimental Setup
  12. Performance Analyses
  13. Comparisons and Discussion
  14. Conclusion

Figures (6)

  • Figure 1: The macro architecture of EfficientViT cai2022efficientvit. Each MBConv consists of two pointwise convolutions (PWConvs) separated by a depthwise convolution (DWConv). Besides, the key component of the EfficientViT module is the lightweight Multi-Scale Attention (MSA).
  • Figure 2: (a) DSConv: Depthwise Convolution followed by Pointwise Convolution. (b) The computation flow of ReLU-based global attention in EfficientViT.
  • Figure 3: (a) RPE works in DW Mode. (b) The micro-architecture of proposed Reconfigurable Processing Element (RPE). (c) RPE works in PW Mode.
  • Figure 4: The overall architecture of our accelerator. The RPE engine is composed of $M$ PE lines with $M\times N$ multipliers, and the MAT engine is configured as $S$ MATs with $S\times T$ multipliers.
  • Figure 5: The proposed time-multiplexed and pipelined (TMP) dataflow.
  • ...and 1 more figures