Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A Quantitative Evaluation of Approximate Softmax Functions for Deep Neural Networks

Anthony Leiva-Valverde, Fabricio Elizondo-Fernández, Luis G. León-Vega, Cristina Meinhardt, Jorge Castro-Godínez

TL;DR

The paper targets efficient softmax inference on resource-constrained FPGAs by evaluating approximate softmax implementations using Taylor-series and LUT-based interpolation. It formalizes the optimization framework, defines the softmax domain $S$ and stability scaling, and compares 3rd-order Taylor vs LUT-based methods using RMSE and hardware metrics. Key findings show that quadratic LUT interpolation achieves the smallest numerical error, while Taylor-based methods offer faster execution and lower resource usage; in LeNet-5 and MobileNet v2, 0.2% accuracy degradation with 14% resource savings and up to a 16.6% accuracy gain with 20% resource savings are observed. The results demonstrate that approximate softmax designs are viable for edge DL inference and hold promise for scaling to larger models, including LLMs.

Abstract

The softmax function is a widely used activation function in the output layers of neural networks, responsible for converting raw scores into class probabilities while introducing essential non-linearity. Implementing Softmax efficiently poses challenges on low-end FPGAs due to limited hardware resources and the computational complexity of exponential and division operations. This work evaluates approximate computing techniques for softmax acceleration using Taylor series and interpolation methods using Look-Up Tables (LUTs). These approximations aim to reduce execution time and resource consumption while maintaining acceptable levels of numerical precision. Our findings show that quadratic interpolation with LUTs yields the lowest numerical error. In contrast, Taylor-based approximations offer significantly better performance in terms of execution time and resource efficiency due to their computational simplicity. When applied to real-world deep learning models such as LeNet-5 and MobileNet v2, the first- and second-order Taylor approximations provided substantial trade-offs between accuracy and resource savings, achieving up to 0.2% accuracy degradation and 14% resource reduction compared to exact implementations. These results highlight the effectiveness of approximate Softmax designs on resource-constrained FPGAs and lay the groundwork for their integration into larger models, including large language models (LLMs).

A Quantitative Evaluation of Approximate Softmax Functions for Deep Neural Networks

TL;DR

The paper targets efficient softmax inference on resource-constrained FPGAs by evaluating approximate softmax implementations using Taylor-series and LUT-based interpolation. It formalizes the optimization framework, defines the softmax domain and stability scaling, and compares 3rd-order Taylor vs LUT-based methods using RMSE and hardware metrics. Key findings show that quadratic LUT interpolation achieves the smallest numerical error, while Taylor-based methods offer faster execution and lower resource usage; in LeNet-5 and MobileNet v2, 0.2% accuracy degradation with 14% resource savings and up to a 16.6% accuracy gain with 20% resource savings are observed. The results demonstrate that approximate softmax designs are viable for edge DL inference and hold promise for scaling to larger models, including LLMs.

Abstract

The softmax function is a widely used activation function in the output layers of neural networks, responsible for converting raw scores into class probabilities while introducing essential non-linearity. Implementing Softmax efficiently poses challenges on low-end FPGAs due to limited hardware resources and the computational complexity of exponential and division operations. This work evaluates approximate computing techniques for softmax acceleration using Taylor series and interpolation methods using Look-Up Tables (LUTs). These approximations aim to reduce execution time and resource consumption while maintaining acceptable levels of numerical precision. Our findings show that quadratic interpolation with LUTs yields the lowest numerical error. In contrast, Taylor-based approximations offer significantly better performance in terms of execution time and resource efficiency due to their computational simplicity. When applied to real-world deep learning models such as LeNet-5 and MobileNet v2, the first- and second-order Taylor approximations provided substantial trade-offs between accuracy and resource savings, achieving up to 0.2% accuracy degradation and 14% resource reduction compared to exact implementations. These results highlight the effectiveness of approximate Softmax designs on resource-constrained FPGAs and lay the groundwork for their integration into larger models, including large language models (LLMs).
Paper Structure (10 sections, 8 equations, 2 figures, 4 tables)

This paper contains 10 sections, 8 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Optimisation framework
  3. Definition
  4. Taylor approximation
  5. LUT-based piece-wise interpolation
  6. Numerical error metric
  7. Standalone Numerical and Performance Evaluation
  8. Resource and Performance Evaluation on Actual Deep Learning Models
  9. Related Work
  10. Conclusion

Figures (2)

  • Figure 1: Piecewise representation by doing eight samples within the domain $S$ and applying a linear interpolation
  • Figure 2: Resource usage and execution time of the softmax accelerators based on 3rd-order Taylor and 64-sample Linear Interpolation processing a 1024 16-bit fixed-point vector. The resource consumption is relative to an AMD Kria KV260.