Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Hyft: A Reconfigurable Softmax Accelerator with Hybrid Numeric Format for both Training and Inference

Tianhua Xia, Sai Qian Zhang

TL;DR

Hyft is proposed, a hardware efficient floating point Softmax accelerator for both training and inference that achieves a remarkable 10X reduction in hardware resource utilization and a 6x reduction in processing latency, all while maintaining a negligible impact on transformer accuracy.

Abstract

The attention mechanism is a pivotal element within the transformer architecture, making a substantial contribution to its exceptional performance. Within this attention mechanism, Softmax is an imperative component that enables the model to assess the degree of correlation between various segments of the input. Yet, prior research has shown that Softmax operations can significantly increase processing latency and energy consumption in the transformer network due to their internal nonlinear operations and data dependencies. In this work, we proposed Hyft, a hardware efficient floating point Softmax accelerator for both training and inference. Hyft aims to reduce the implementation cost of different nonlinear arithmetic operations within softmax by adaptively converting intermediate results into the most suitable numeric format for each specific operation, leading to reconfigurable accelerator with hybrid numeric format. The evaluation results highlight that Hyft achieves a remarkable 10x reduction in hardware resource utilization and a 6x reduction in processing latency, all while maintaining a negligible impact on transformer accuracy.

Hyft: A Reconfigurable Softmax Accelerator with Hybrid Numeric Format for both Training and Inference

TL;DR

Hyft is proposed, a hardware efficient floating point Softmax accelerator for both training and inference that achieves a remarkable 10X reduction in hardware resource utilization and a 6x reduction in processing latency, all while maintaining a negligible impact on transformer accuracy.

Abstract

The attention mechanism is a pivotal element within the transformer architecture, making a substantial contribution to its exceptional performance. Within this attention mechanism, Softmax is an imperative component that enables the model to assess the degree of correlation between various segments of the input. Yet, prior research has shown that Softmax operations can significantly increase processing latency and energy consumption in the transformer network due to their internal nonlinear operations and data dependencies. In this work, we proposed Hyft, a hardware efficient floating point Softmax accelerator for both training and inference. Hyft aims to reduce the implementation cost of different nonlinear arithmetic operations within softmax by adaptively converting intermediate results into the most suitable numeric format for each specific operation, leading to reconfigurable accelerator with hybrid numeric format. The evaluation results highlight that Hyft achieves a remarkable 10x reduction in hardware resource utilization and a 6x reduction in processing latency, all while maintaining a negligible impact on transformer accuracy.
Paper Structure (17 sections, 12 equations, 5 figures, 3 tables)

This paper contains 17 sections, 12 equations, 5 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Background and Motivation
  3. Numeric Format
  4. Computation in Transformer
  5. Softmax Function
  6. Related Work
  7. Hyft Architecture
  8. Parameterized Input Pre-Processr
  9. Hybrid Exponent Unit
  10. Hybrid Adder Tree
  11. Division/Multiplication Unit
  12. Softmax Backpropagation For Training
  13. Hyft Vector Processor
  14. Experimental Results
  15. Accuracy Evaluation
  16. ...and 2 more sections

Figures (5)

  • Figure 1: (a) Percentage of Softmax computation runtime in BigBird. (b) Fixed and floating-point number formats.
  • Figure 2: Forward propagation data path of Hyft.
  • Figure 3: (a) Input Pre-processor and (b) Hybrid Exponent Unit. The floating-point and fixed-point data paths are highlighted in green and red, respectively. The control signal are highlighted in black.
  • Figure 4: (a) Hybrid adder tree and (b) division-multiplication unit.
  • Figure 5: Energy analysis for Hyft across multiple input sequence lengths.