Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Low Power Vision Transformer Accelerator with Hardware-Aware Pruning and Optimized Dataflow

Ching-Lin Hsiung, Tian-Sheuan Chang

TL;DR

This work tackles the bottleneck trade-off in vision transformers by shifting focus from self-attention to the FFN, and presents a hardware-aware co-design that combines hardware-friendly dynamic token pruning, a ReLU activation, and dynamic FFN2 pruning. The row-wise dataflow and integrated pruning modules minimize data movement and off-chip memory traffic, enabling end-to-end acceleration on a 28 nm design with 496K NAND gates and 232 KB SRAM. The approach achieves a 61.5% reduction in MACs with less than 2% accuracy loss, reduces external memory fetch by up to 22.7%, and delivers 1024 GOPS peak throughput at 1 GHz with 2.31 TOPS/W energy efficiency. Overall, the paper demonstrates a practical, low-power ViT accelerator that leverages hardware-aware pruning and dataflow optimizations to enable efficient embedded vision processing.

Abstract

Current transformer accelerators primarily focus on optimizing self-attention due to its quadratic complexity. However, this focus is less relevant for vision transformers with short token lengths, where the Feed-Forward Network (FFN) tends to be the dominant computational bottleneck. This paper presents a low power Vision Transformer accelerator, optimized through algorithm-hardware co-design. The model complexity is reduced using hardware-friendly dynamic token pruning without introducing complex mechanisms. Sparsity is further improved by replacing GELU with ReLU activations and employing dynamic FFN2 pruning, achieving a 61.5\% reduction in operations and a 59.3\% reduction in FFN2 weights, with an accuracy loss of less than 2\%. The hardware adopts a row-wise dataflow with output-oriented data access to eliminate data transposition, and supports dynamic operations with minimal area overhead. Implemented in TSMC's 28nm CMOS technology, our design occupies 496.4K gates and includes a 232KB SRAM buffer, achieving a peak throughput of 1024 GOPS at 1GHz, with an energy efficiency of 2.31 TOPS/W and an area efficiency of 858.61 GOPS/mm2.

Low Power Vision Transformer Accelerator with Hardware-Aware Pruning and Optimized Dataflow

TL;DR

This work tackles the bottleneck trade-off in vision transformers by shifting focus from self-attention to the FFN, and presents a hardware-aware co-design that combines hardware-friendly dynamic token pruning, a ReLU activation, and dynamic FFN2 pruning. The row-wise dataflow and integrated pruning modules minimize data movement and off-chip memory traffic, enabling end-to-end acceleration on a 28 nm design with 496K NAND gates and 232 KB SRAM. The approach achieves a 61.5% reduction in MACs with less than 2% accuracy loss, reduces external memory fetch by up to 22.7%, and delivers 1024 GOPS peak throughput at 1 GHz with 2.31 TOPS/W energy efficiency. Overall, the paper demonstrates a practical, low-power ViT accelerator that leverages hardware-aware pruning and dataflow optimizations to enable efficient embedded vision processing.

Abstract

Current transformer accelerators primarily focus on optimizing self-attention due to its quadratic complexity. However, this focus is less relevant for vision transformers with short token lengths, where the Feed-Forward Network (FFN) tends to be the dominant computational bottleneck. This paper presents a low power Vision Transformer accelerator, optimized through algorithm-hardware co-design. The model complexity is reduced using hardware-friendly dynamic token pruning without introducing complex mechanisms. Sparsity is further improved by replacing GELU with ReLU activations and employing dynamic FFN2 pruning, achieving a 61.5\% reduction in operations and a 59.3\% reduction in FFN2 weights, with an accuracy loss of less than 2\%. The hardware adopts a row-wise dataflow with output-oriented data access to eliminate data transposition, and supports dynamic operations with minimal area overhead. Implemented in TSMC's 28nm CMOS technology, our design occupies 496.4K gates and includes a 232KB SRAM buffer, achieving a peak throughput of 1024 GOPS at 1GHz, with an energy efficiency of 2.31 TOPS/W and an area efficiency of 858.61 GOPS/mm2.
Paper Structure (27 sections, 3 equations, 22 figures, 4 tables)

This paper contains 27 sections, 3 equations, 22 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Proposed Method
  3. Hardware-Friendly Dynamic Token Pruning
  4. Hardware-Friendly Activation
  5. Dynamic FFN2 Weight Pruning
  6. Observation
  7. Proposed FFN2 Weight Pruning
  8. Algorithm-Hardware Co-Design Principles
  9. Hardware Design
  10. PE
  11. Dataflow and detailed mapping
  12. Fully connected operations
  13. FFN
  14. Self-attention
  15. Token Pruning
  16. ...and 12 more sections

Figures (22)

  • Figure 1: Distribution analysis of the DeiT-small model
  • Figure 2: Inference process within a single encoder block
  • Figure 3: Visualization of the token pruning process
  • Figure 4: Histogram of FFN intermediate results before and after ReLU finetuning. (a) Before ReLU (b) After ReLU
  • Figure 5: Distribution of FFN post-activation matrix values
  • ...and 17 more figures