HG-PIPE: Vision Transformer Acceleration with Hybrid-Grained Pipeline

TL;DR

HG-PIPE presents a hybrid-grained pipelined FPGA accelerator for Vision Transformers that balances memory access, buffer costs, and compute by combining deep K/V buffers with a Transpose module and finely-tuned OS dataflow. It introduces LUT-based MAC units, Power-of-Two index approximation, and GeLU-ReQuant fusion to dramatically reduce DSP usage while preserving accuracy, achieving up to 7118 images/s on VCK190 and outperforming comparable GPUs and FPGA baselines. The work provides extensive engineering optimizations—including BRAM-efficient tiling, asynchronous decentralized control, and joint table calibrations—that collectively push ViT throughput and energy efficiency on edge devices. Practically, HG-PIPE enables real-time ViT inference on a single FPGA device with strong resource efficiency, making high-performance, low-latency vision processing more accessible for embedded systems.

Abstract

Vision Transformer (ViT) acceleration with field programmable gate array (FPGA) is promising but challenging. Existing FPGA-based ViT accelerators mainly rely on temporal architectures, which process different operators by reusing the same hardware blocks and suffer from extensive memory access overhead. Pipelined architectures, either coarse-grained or fine-grained, unroll the ViT computation spatially for memory access efficiency. However, they usually suffer from significant hardware resource constraints and pipeline bubbles induced by the global computation dependency of ViT. In this paper, we introduce HG-PIPE, a pipelined FPGA accelerator for high-throughput and low-latency ViT processing. HG-PIPE features a hybrid-grained pipeline architecture to reduce on-chip buffer cost and couples the computation dataflow and parallelism design to eliminate the pipeline bubbles. HG-PIPE further introduces careful approximations to implement both linear and non-linear operators with abundant Lookup Tables (LUTs), thus alleviating resource constraints. On a ZCU102 FPGA, HG-PIPE achieves 2.78 times better throughput and 2.52 times better resource efficiency than the prior-art accelerators, e.g., AutoViTAcc. With a VCK190 FPGA, HG-PIPE realizes end-to-end ViT acceleration on a single device and achieves 7118 images/s, which is 2.81 times faster than a V100 GPU.

Figures (13)

• Figure 1: Roofline model for FPGA-based ViT acceleration.
• Figure 2: Compare coarse-grained and fine-grained pipeline.
• Figure 3: The challenges of ViT hardware acceleration.
• Figure 4: Overview of the HG-PIPE accelerator design.
• Figure 5: The dataflow design of the MHA module.