Resource-Efficient Neural Networks for Embedded Systems

TL;DR

This work surveys resource-efficient neural networks for embedded systems, focusing on three main axes: quantized neural networks, network pruning, and structural efficiency. It connects algorithmic techniques (including STE-based training, Bayesian methods, weight sharing, distillation, and NAS) with hardware considerations across CPUs, GPUs, and FPGAs, and provides experimental insights on CIFAR-10/100. The findings highlight the importance of mixed-precision quantization, structured pruning, and hardware-aware NAS to balance accuracy with memory, latency, and energy. Practically, the study underlines that hardware characteristics largely dictate which compression and design strategies yield meaningful throughput gains while preserving predictive performance.

Abstract

While machine learning is traditionally a resource intensive task, embedded systems, autonomous navigation, and the vision of the Internet of Things fuel the interest in resource-efficient approaches. These approaches aim for a carefully chosen trade-off between performance and resource consumption in terms of computation and energy. The development of such approaches is among the major challenges in current machine learning research and key to ensure a smooth transition of machine learning technology from a scientific environment with virtually unlimited computing resources into everyday's applications. In this article, we provide an overview of the current state of the art of machine learning techniques facilitating these real-world requirements. In particular, we focus on resource-efficient inference based on deep neural networks (DNNs), the predominant machine learning models of the past decade. We give a comprehensive overview of the vast literature that can be mainly split into three non-mutually exclusive categories: (i) quantized neural networks, (ii) network pruning, and (iii) structural efficiency. These techniques can be applied during training or as post-processing, and they are widely used to reduce the computational demands in terms of memory footprint, inference speed, and energy efficiency. We also briefly discuss different concepts of embedded hardware for DNNs and their compatibility with machine learning techniques as well as potential for energy and latency reduction. We substantiate our discussion with experiments on well-known benchmark data sets using compression techniques (quantization, pruning) for a set of resource-constrained embedded systems, such as CPUs, GPUs and FPGAs. The obtained results highlight the difficulty of finding good trade-offs between resource efficiency and prediction quality.

Figures (8)

• Figure 1: Aspects of resource-efficient machine learning models.
• Figure 2: A simplified building block of a DNN using the straight-through gradient estimator (STE). $Q$ denotes some arbitrary piecewise constant quantization function and id denotes the identity function which simply passes the gradient on during backpropagation. In the forward pass, the solid red line is followed which passes the two piecewise constant functions $Q$ and $\mathop{\mathrm{sign}}\limits$ whose gradient is zero almost everywhere (red boxes). During backpropagation, the dashed green line is followed which avoids these piecewise constant functions and instead only passes differentiable functions (green boxes)---in particular, the functions id and $\tanh$ whose shapes are similar to $Q$ and $\mathop{\mathrm{sign}}\limits$ but whose gradient is non-zero. This allows us to obtain an approximate non-zero gradient for the real-valued parameters $\mathbf{W}^l$ (blue circle) which are subsequently updated with SGD.
• Figure 3: Comparison of several popular quantization approaches (see Section \ref{['sec:literature_quantization']}) using the DenseNet-BC-100 architecture trained on the CIFAR-100 data set. The horizontal red line shows the error of the real-valued baseline. Quantization is performed using different bit widths in three different modes: activation-only (blue), weight-only (green), and combined weight and activation quantization (purple).
• Figure 4: Performance metrics of several pruning structures and DNN architectures on CIFAR-10 for WRNs and DenseNets.
• Figure 5: Throughput-accuracy trade-off of pruned and quantized WRN models on the CIFAR-10 task for an ARM CPU.