ONE-SA: Enabling Nonlinear Operations in Systolic Arrays for Efficient and Flexible Neural Network Inference

TL;DR

This work tackles the rigidity of traditional DNN accelerators by embedding nonlinear computation inside a systolic array using capped piecewise linear (CPWL) approximations. Nonlinear functions are computed through Intermediate Parameter Fetching (IPF) and Matrix Hadamard Product (MHP), preserving the linear GEMM path while enabling versatile network inference. The design introduces an L3 buffer for intermediate parameters and a diagonal Processing Element (PE) scheme with Transmission PEs to realize MHP, achieving near-ASIC efficiency with much greater model flexibility. Experiments on FPGA show negligible accuracy loss for common DNNs and substantial speedups over CPUs/GPUs, with competitive performance relative to application-specific accelerators.

Abstract

The computation and memory-intensive nature of DNNs limits their use in many mobile and embedded contexts. Application-specific integrated circuit (ASIC) hardware accelerators employ matrix multiplication units (such as the systolic arrays) and dedicated nonlinear function units to speed up DNN computations. A close examination of these ASIC accelerators reveals that the designs are often specialized and lack versatility across different networks, especially when the networks have different types of computation. In this paper, we introduce a novel systolic array architecture, which is capable of executing nonlinear functions. By encompassing both inherent linear and newly enabled nonlinear functions within the systolic arrays, the proposed architecture facilitates versatile network inferences, substantially enhancing computational power and energy efficiency. Experimental results show that employing this systolic array enables seamless execution of entire DNNs, incurring only a negligible loss in the network inference accuracy. Furthermore, assessment and evaluation with FPGAs reveal that integrating nonlinear computation capacity into a systolic array does not introduce extra notable (less than 1.5%) block memory memories (BRAMs), look-up-tables (LUTs), or digital signal processors (DSPs) but a mere 13.3% - 24.1% more flip flops (FFs). In comparison to existing methodologies, executing the networks with the proposed systolic array, which enables the flexibility of different network models, yields up to 25.73x, 5.21x, and 1.54x computational efficiency when compared to general-purpose CPUs, GPUs, and SoCs respectively, while achieving comparable (83.4% - 135.8%) performance with the conventional accelerators which are designed for specific neural network models.

Figures (10)

• Figure 1: The computations in classic neural network models.
• Figure 2: A classic systolic array architecture with 3x3 PEs.
• Figure 3: Capped piecewise linearization for nonlinear operations.
• Figure 4: Proposed systolic array architecture with nonlinear operations.
• Figure 5: The microarchitecture of L3 buffers data addressing.