Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

GRAU: Generic Reconfigurable Activation Unit Design for Neural Network Hardware Accelerators

Yuhao Liu, Salim Ullah, Akash Kumar

TL;DR

This work proposes a reconfigurable activation hardware, GRAU, based on piecewise linear fitting, where the segment slopes are approximated by powers of two, achieving higher hardware efficiency, flexibility, and scalability.

Abstract

With the continuous growth of neural network scales, low-precision quantization is widely used in edge accelerators. Classic multi-threshold activation hardware requires 2^n thresholds for n-bit outputs, causing a rapid increase in hardware cost as precision increases. We propose a reconfigurable activation hardware, GRAU, based on piecewise linear fitting, where the segment slopes are approximated by powers of two. Our design requires only basic comparators and 1-bit right shifters, supporting mixed-precision quantization and nonlinear functions such as SiLU. Compared with multi-threshold activators, GRAU reduces LUT consumption by over 90%, achieving higher hardware efficiency, flexibility, and scalability.

GRAU: Generic Reconfigurable Activation Unit Design for Neural Network Hardware Accelerators

TL;DR

This work proposes a reconfigurable activation hardware, GRAU, based on piecewise linear fitting, where the segment slopes are approximated by powers of two, achieving higher hardware efficiency, flexibility, and scalability.

Abstract

With the continuous growth of neural network scales, low-precision quantization is widely used in edge accelerators. Classic multi-threshold activation hardware requires 2^n thresholds for n-bit outputs, causing a rapid increase in hardware cost as precision increases. We propose a reconfigurable activation hardware, GRAU, based on piecewise linear fitting, where the segment slopes are approximated by powers of two. Our design requires only basic comparators and 1-bit right shifters, supporting mixed-precision quantization and nonlinear functions such as SiLU. Compared with multi-threshold activators, GRAU reduces LUT consumption by over 90%, achieving higher hardware efficiency, flexibility, and scalability.
Paper Structure (16 sections, 4 figures, 4 tables)

This paper contains 16 sections, 4 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Motivation
  3. Exponential hardware scaling with precision
  4. Inefficiency in mixed-precision quantization
  5. Inability to represent non-monotonic activations
  6. Related Works
  7. Contribution
  8. Organization
  9. Implementation
  10. PoT and APoT Approximated Activation Functions for GRAU Hardware
  11. Hardware Implementation of GRAU
  12. Evaluation
  13. Hardware Resource Consumption
  14. Latency
  15. Total Delay, Power, ADP, and PDP
  16. ...and 1 more sections

Figures (4)

  • Figure 1: Correct 2-bit quantization of Multi-Threshold unit (left) in Sigmoid and the mistake of Multi-Threshold unit in non-monotonically increasing function (right)
  • Figure 2: Comparing the original nonlinear function, PWLF approximated function, PoT approximated PWLF function, and APoT approximated PWLF function
  • Figure 3: Encoding of segment slopes for PoT-PWLF (down) and APoT-PWLF (up) approximation
  • Figure 5: Serialized hardware implementation of Generic Activation Unit PoT-PWLF and APoT-PWLF shifter unit