EIE: Efficient Inference Engine on Compressed Deep Neural Network

TL;DR

The paper tackles the high energy and memory demands of large DNNs by designing EIE, an accelerator that operates directly on compressed, sparse networks with weight sharing. EIE uses a CSC-based encoding, dynamic activation sparsity, and a scalable PE array with Leading Non-Zero Detection to perform sparse matrix-vector multiplication efficiently in 16-bit fixed-point. The approach delivers dramatic improvements over CPU/GPU baselines (up to 189× throughput and thousands of times in energy efficiency) and demonstrates real-time FC-layer inference on AlexNet-scale models within powered-on-chip SRAM. This work enables large, state-of-the-art DNNs to run on embedded hardware with stringent resource constraints and latency requirements.

Abstract

State-of-the-art deep neural networks (DNNs) have hundreds of millions of connections and are both computationally and memory intensive, making them difficult to deploy on embedded systems with limited hardware resources and power budgets. While custom hardware helps the computation, fetching weights from DRAM is two orders of magnitude more expensive than ALU operations, and dominates the required power. Previously proposed 'Deep Compression' makes it possible to fit large DNNs (AlexNet and VGGNet) fully in on-chip SRAM. This compression is achieved by pruning the redundant connections and having multiple connections share the same weight. We propose an energy efficient inference engine (EIE) that performs inference on this compressed network model and accelerates the resulting sparse matrix-vector multiplication with weight sharing. Going from DRAM to SRAM gives EIE 120x energy saving; Exploiting sparsity saves 10x; Weight sharing gives 8x; Skipping zero activations from ReLU saves another 3x. Evaluated on nine DNN benchmarks, EIE is 189x and 13x faster when compared to CPU and GPU implementations of the same DNN without compression. EIE has a processing power of 102GOPS/s working directly on a compressed network, corresponding to 3TOPS/s on an uncompressed network, and processes FC layers of AlexNet at 1.88x10^4 frames/sec with a power dissipation of only 600mW. It is 24,000x and 3,400x more energy efficient than a CPU and GPU respectively. Compared with DaDianNao, EIE has 2.9x, 19x and 3x better throughput, energy efficiency and area efficiency.

Figures (13)

• Figure 1: Efficient inference engine that works on the compressed deep neural network model for machine learning applications.
• Figure 2: Matrix $W$ and vectors $a$ and $b$ are interleaved over 4 PEs. Elements of the same color are stored in the same PE.
• Figure 3: Memory layout for the relative indexed, indirect weighted and interleaved CSC format, corresponding to PE$_0$ in Figure \ref{['fig:matrix']}.
• Figure 4: (a) The architecture of Leading Non-zero Detection Node. (b) The architecture of Processing Element.
• Figure 5: Layout of one PE in EIE under TSMC 45nm process.