Efficient CNN Inference on Ultra-Low-Power MCUs via Saturation-Aware Convolution

TL;DR

The paper tackles the latency and energy bottlenecks of quantized CNN inference on ultra-low-power MCUs by revealing and exploiting effectless computations caused by neuron saturation. It introduces saturation-aware convolution, which combines zero-error omission using bounds $d_{min,i}$/$d_{max,i}$, reordering by weight magnitude $|w_i|$, and compile-time profiling to place saturation checks, all implemented as an extension to TinyEngine within MCUNet. The approach yields substantial speedups (up to 24%) and energy savings (up to 23%), with strictly zero changes to network accuracy, on seven open-source CNN workloads run on a Cortex-M0+ MCU; minor memory overhead is incurred due to redirection arrays and saturation-check data. This work enables practical, energy-efficient CNN deployment on microcontrollers by exploiting quantization-induced saturation without modifying the models.

Abstract

Deploying lightweight CNN inference tasks on ultra-low-power MCUs is often not limited by space constraint, thanks to the compact size of models, yet inference latency is crucial for preserving energy. We reveal that quantized CNN inference on ultra-low-power MCUs executes unnecessary computations in neurons that produce saturated output values: often times, these neurons still produce the correct output value without fully completing the computation, since the neuron value is too extreme and is eventually systematically clamped at the boundaries allowed by the neuron. We show that with carefully designed condition checks, it is possible to identify and skip these unnecessary computations without impacting the neuron output. Based on this, we present saturation-aware convolution: an inference technique whereby computations in convolution kernels are executed in an altered order to induce earlier saturation, and saturation checks are inserted to omit unnecessary computations. We integrate our implementation into MCUNet's TinyEngine, the state-of-the-art neural network code generation and inference framework, and conduct experiments on a Cortex-M0+ MCU. The result based on 7 open-source CNN models displays up to 24% inference time saving, with strictly zero impact on neural network accuracy.

Figures (9)

• Figure 1: An example of the computation trace of a convolution operation, where the output neuron's final value will be saturated to int8. Because any smaller value will be clamped to -128, removing the last 10 computations from this convolution operation will not change the output neuron's value.
• Figure 2: Pct. of computations with no effect on neuron outputs in convolutional and fully-connected layers in cnn workloads.
• Figure 3: A counter example in the same layer, but a different neuron as Figure \ref{['fig:acc_change_no_errorbar']}. The intermediate accumulated value going out of bound does not guarantee a saturated final output.
• Figure 4: The computation trace of the convolution operation from the same example as Figure 1. The error bars indicate the maximum and minimum possible final accumulated value. This is calculated by assuming all future computations generate the most extreme results allowed by int8.
• Figure 5: The percentage of effectless computations that can be omitted with strictly zero error in convolutional and fully-connected layers, with and without computation reordering.