msf-CNN: Patch-based Multi-Stage Fusion with Convolutional Neural Networks for TinyML

TL;DR

This work addresses the challenge of running CNNs on memory-constrained microcontrollers by introducing msf-CNN, a patch-based multi-stage fusion framework. It represents CNNs as inverted dataflow DAGs where edges encode RAM and MAC costs, including potential fusion blocks, and optimizes fusion configurations via offline graph-based shortest-path algorithms under RAM and compute constraints. The authors provide a pruning-enabled solution with polynomial-time guarantees, rewrite global pooling and dense layers to further reduce RAM, and implement a full open-source pipeline on MCUs (ARM Cortex-M, RISC-V, ESP32). Experimental results show up to roughly 50%–87% peak RAM reductions compared with prior methods, with corresponding latency trade-offs, demonstrating flexible memory-latency tuning for TinyML deployments. The approach is hardware-agnostic and extensible to other CPU architectures and accelerators, enabling broader adoption of memory-optimized CNN inference on ultra-low-resource devices.

Abstract

AI spans from large language models to tiny models running on microcontrollers (MCUs). Extremely memory-efficient model architectures are decisive to fit within an MCU's tiny memory budget e.g., 128kB of RAM. However, inference latency must remain small to fit real-time constraints. An approach to tackle this is patch-based fusion, which aims to optimize data flows across neural network layers. In this paper, we introduce msf-CNN, a novel technique that efficiently finds optimal fusion settings for convolutional neural networks (CNNs) by walking through the fusion solution space represented as a directed acyclic graph. Compared to previous work on CNN fusion for MCUs, msf-CNN identifies a wider set of solutions. We published an implementation of msf-CNN running on various microcontrollers (ARM Cortex-M, RISC-V, ESP32). We show that msf-CNN can achieve inference using 50% less RAM compared to the prior art (MCUNetV2 and StreamNet). We thus demonstrate how msf-CNN offers additional flexibility for system designers.

Figures (4)

• Figure 1: (a) Overview of msf-CNN. The convolutional layers are fused into several fusion blocks based on the optimal setting found by optimizer. We let global Pooling and dense layers compute the outputs iteratively to further squeeze RAM usage. (b) The neural network is modeled as a DAG. Nodes $v_n$ denote the tensors that are produced and consumed by the operators or possible fusion blocks. Edges $e_1, \dots, e_4$ represent individual operators, while edge $e_5, e_6$ represent two candidate fusion blocks. Edges are annotated with the RAM usage and MAC amounts of their corresponding operators and fusion blocks.
• Figure 2: Comparison of common and iterative global pooling.
• Figure 3: Comparison of common and iterative dense layer. The columns of the weight matrix are denoted as $w(n)$.
• Figure 4: Trade-off between RAM and latency of different optimal fusion settings on Nucleo-f767zi. P1: Minimize RAM s.t. compute cost limit. P2: Minimize compute cost s.t. RAM limit. For more detailed results, please refer to \ref{['tbl:impact_mem_and_mac']} in \ref{['appdix:exp_details']}.