Benchmarking Energy and Latency in TinyML: A Novel Method for Resource-Constrained AI

TL;DR

The paper tackles the challenge of benchmarking TinyML on resource-constrained MCUs by introducing a phase-aware energy and latency measurement methodology that directly measures core energy using a shunt resistor and isolates pre/inference/post phases. It critiques MLPerf Tiny's limitations and proposes EDP and rEDP as unified efficiency metrics to enable cross-platform comparisons. It validates the approach on STM32N6 with H-Perf and L-Perf configurations across four MLPerf Tiny models, demonstrating energy savings and identifying pre-inference as a bottleneck. The methodology enables simultaneous measurement of energy, latency, and accuracy within a single firmware, supporting statistically robust, repeatable comparisons across platforms.

Abstract

The rise of IoT has increased the need for on-edge machine learning, with TinyML emerging as a promising solution for resource-constrained devices such as MCU. However, evaluating their performance remains challenging due to diverse architectures and application scenarios. Current solutions have many non-negligible limitations. This work introduces an alternative benchmarking methodology that integrates energy and latency measurements while distinguishing three execution phases pre-inference, inference, and post-inference. Additionally, the setup ensures that the device operates without being powered by an external measurement unit, while automated testing can be leveraged to enhance statistical significance. To evaluate our setup, we tested the STM32N6 MCU, which includes a NPU for executing neural networks. Two configurations were considered: high-performance and Low-power. The variation of the EDP was analyzed separately for each phase, providing insights into the impact of hardware configurations on energy efficiency. Each model was tested 1000 times to ensure statistically relevant results. Our findings demonstrate that reducing the core voltage and clock frequency improve the efficiency of pre- and post-processing without significantly affecting network execution performance. This approach can also be used for cross-platform comparisons to determine the most efficient inference platform and to quantify how pre- and post-processing overhead varies across different hardware implementations.

Figures (6)

• Figure 1: Block diagram of the measurement setup. The measurement instrument measures the voltage drop across the shunt resistor from Ch1 and Ch2, and monitors two trigger signals from Ch3 and Ch4 to distinguish four different phases (Pre-, Post-, inference, and idle). The PC communicates with the DUT through the serial interface. Meanwhile, the MCU automatically transmits classifier outputs to the Host PC, enabling accuracy measurements during a subsequent phase and setting or resetting one of the reference signals during each phase
• Figure 2: Development kit STM32N6570-DK used for the measurements, featuring the N6 MCU and multiple peripherals
• Figure 3: Measurement setup showing the STM32N6570-DK development kit connected to the MSO64B oscilloscope. On the left, two probes measure the voltage drop across $R_{shunt}$ of the STM32N6 core, while on the right, two probes detect the two trigger signals.
• Figure 4: Oscilloscope trace of the energy consumption during an inference process, segmented into Pre-Inference (orange), Inference (green), and Post-Inference (purple) phases. The blue trace shows the voltage drop across $R_\text{shunt}$, while the red dashed line indicates cumulative energy consumption. Circular and square markers denote the start and end of the energy measurement interval, respectively.
• Figure 5: Comparison of execution time (a) and energy consumption (b) across the three phases under H-Perf (dashed line) and L-Perf (continuous lines) configurations. The execution time (latency) is the same for both pre- and post- inference phases since the CPU is speed is the same, while there is a reduction in energy consumption primarily due to voltage core reduction