PEANO-ViT: Power-Efficient Approximations of Non-Linearities in Vision Transformers

TL;DR

PEANO-ViT tackles the challenge of deploying Vision Transformers on power-constrained FPGAs by replacing key non-linear blocks with hardware-friendly approximations: division-free layer normalization, Padé-based exponential for softmax, a multi-scale reciprocal approximation, and a piecewise linear GELU. The approach delivers minimal accuracy loss while achieving substantial power and resource savings, validated on ImageNet-1K across ViT-based models with an FPGA implementation. Its flexibility—tunable LN, softmax precision, and GELU segmentation—allows tailored accuracy, latency, and energy trade-offs for hardware-limited deployments. This work significantly eases practical, energy-efficient ViT deployment on resource-constrained hardware and sets a framework for hardware-aware neural network design.

Abstract

The deployment of Vision Transformers (ViTs) on hardware platforms, specially Field-Programmable Gate Arrays (FPGAs), presents many challenges, which are mainly due to the substantial computational and power requirements of their non-linear functions, notably layer normalization, softmax, and Gaussian Error Linear Unit (GELU). These critical functions pose significant obstacles to efficient hardware implementation due to their complex mathematical operations and the inherent resource count and architectural limitations of FPGAs. PEANO-ViT offers a novel approach to streamlining the implementation of the layer normalization layer by introducing a division-free technique that simultaneously approximates the division and square root function. Additionally, PEANO-ViT provides a multi-scale division strategy to eliminate division operations in the softmax layer, aided by a Pade-based approximation for the exponential function. Finally, PEANO-ViT introduces a piece-wise linear approximation for the GELU function, carefully designed to bypass the computationally intensive operations associated with GELU. In our comprehensive evaluations, PEANO-ViT exhibits minimal accuracy degradation (<= 0.5% for DeiT-B) while significantly enhancing power efficiency, achieving improvements of 1.91x, 1.39x, 8.01x for layer normalization, softmax, and GELU, respectively. This improvement is achieved through substantial reductions in DSP, LUT, and register counts for these non-linear operations. Consequently, PEANO-ViT enables efficient deployment of Vision Transformers on resource- and power-constrained FPGA platforms.

Figures (2)

• Figure 1: Comparison of standard functions with our approximations.
• Figure 2: Overall FPGA implementation of PEANO-ViT