Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accelerating TinyML Inference on Microcontrollers through Approximate Kernels

Giorgos Armeniakos, Georgios Mentzos, Dimitrios Soudris

TL;DR

The paper addresses the latency and memory constraints of CNN inference on MCU-based TinyML systems. It proposes a cooperative framework that combines layer-based kernel unpacking, compile-time fixed-weight optimization, and offline significance-aware computation skipping guided by design space exploration to identify Pareto-optimal latency-accuracy trade-offs. The framework reduces flash usage and runtime overhead, enabling faster inference with minimal or zero accuracy loss on CIFAR-10 CNNs evaluated on an STM32 MCU, and it outperforms or matches state-of-the-art libraries and compilers in several scenarios. This work demonstrates that approximate computing, when tightly integrated with MCU kernels, can widen the feasible model complexity and enable real-time TinyML deployments on constrained devices.

Abstract

The rapid growth of microcontroller-based IoT devices has opened up numerous applications, from smart manufacturing to personalized healthcare. Despite the widespread adoption of energy-efficient microcontroller units (MCUs) in the Tiny Machine Learning (TinyML) domain, they still face significant limitations in terms of performance and memory (RAM, Flash). In this work, we combine approximate computing and software kernel design to accelerate the inference of approximate CNN models on MCUs. Our kernel-based approximation framework firstly unpacks the operands of each convolution layer and then conducts an offline calculation to determine the significance of each operand. Subsequently, through a design space exploration, it employs a computation skipping approximation strategy based on the calculated significance. Our evaluation on an STM32-Nucleo board and 2 popular CNNs trained on the CIFAR-10 dataset shows that, compared to state-of-the-art exact inference, our Pareto optimal solutions can feature on average 21% latency reduction with no degradation in Top-1 classification accuracy, while for lower accuracy requirements, the corresponding reduction becomes even more pronounced.

Accelerating TinyML Inference on Microcontrollers through Approximate Kernels

TL;DR

The paper addresses the latency and memory constraints of CNN inference on MCU-based TinyML systems. It proposes a cooperative framework that combines layer-based kernel unpacking, compile-time fixed-weight optimization, and offline significance-aware computation skipping guided by design space exploration to identify Pareto-optimal latency-accuracy trade-offs. The framework reduces flash usage and runtime overhead, enabling faster inference with minimal or zero accuracy loss on CIFAR-10 CNNs evaluated on an STM32 MCU, and it outperforms or matches state-of-the-art libraries and compilers in several scenarios. This work demonstrates that approximate computing, when tightly integrated with MCU kernels, can widen the feasible model complexity and enable real-time TinyML deployments on constrained devices.

Abstract

The rapid growth of microcontroller-based IoT devices has opened up numerous applications, from smart manufacturing to personalized healthcare. Despite the widespread adoption of energy-efficient microcontroller units (MCUs) in the Tiny Machine Learning (TinyML) domain, they still face significant limitations in terms of performance and memory (RAM, Flash). In this work, we combine approximate computing and software kernel design to accelerate the inference of approximate CNN models on MCUs. Our kernel-based approximation framework firstly unpacks the operands of each convolution layer and then conducts an offline calculation to determine the significance of each operand. Subsequently, through a design space exploration, it employs a computation skipping approximation strategy based on the calculated significance. Our evaluation on an STM32-Nucleo board and 2 popular CNNs trained on the CIFAR-10 dataset shows that, compared to state-of-the-art exact inference, our Pareto optimal solutions can feature on average 21% latency reduction with no degradation in Top-1 classification accuracy, while for lower accuracy requirements, the corresponding reduction becomes even more pronounced.
Paper Structure (7 sections, 3 equations, 2 figures, 2 tables)

This paper contains 7 sections, 3 equations, 2 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Cooperative Approximation Framework for Inference Optimization
  3. Customized kernels for NN deployment
  4. Layer-based code unpacking
  5. Significance-aware skipping
  6. Experimental Results and Analysis
  7. Conclusion

Figures (2)

  • Figure 1: Abstract overview of our framework
  • Figure 2: Pareto space between accuracy and normalized MAC unit reduction is depicted for our computation skipping approach within all convolution layers for AlexNet (a) and LeNet (b).