Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

MCU-MixQ: A HW/SW Co-optimized Mixed-precision Neural Network Design Framework for MCUs

Junfeng Gong, Cheng Liu, Long Cheng, Huawei Li, Xiaowei Li

TL;DR

This work addresses the challenge of deploying mixed-precision neural networks on resource-constrained MCUs by proposing MCU-MixQ, a HW/SW co-design framework that jointly optimizes quantization and operator implementation. It combines SIMD-based low-bitwidth operator packing (SLBC) with a differentiable NAS-based hardware-aware quantization explorer, and integrates with TinyEngine for MCU deployment. A performance predictor guides the NAS to balance accuracy and runtime efficiency, while adaptive packing and reordering further improve SIMD utilization. Empirical results show significant speedups over CMix-NN and MCUNet under the same constraints, validating the practicality of co-optimizing both quantization and operator design for MCU inference.

Abstract

Mixed-precision neural network (MPNN) that utilizes just enough data width for the neural network processing is an effective approach to meet the stringent resources constraints including memory and computing of MCUs. Nevertheless, there is still a lack of sub-byte and mixed-precision SIMD operations in MCU-class ISA and the limited computing capability of MCUs remains underutilized, which further aggravates the computing bound encountered in neural network processing. As a result, the benefits of MPNNs cannot be fully unleashed. In this work, we propose to pack multiple low-bitwidth arithmetic operations within a single instruction multiple data (SIMD) instructions in typical MCUs, and then develop an efficient convolution operator by exploring both the data parallelism and computing parallelism in convolution along with the proposed SIMD packing. Finally, we further leverage Neural Architecture Search (NAS) to build a HW/SW co-designed MPNN design framework, namely MCU-MixQ. This framework can optimize both the MPNN quantization and MPNN implementation efficiency, striking an optimized balance between neural network performance and accuracy. According to our experiment results, MCU-MixQ achieves 2.1$\times$ and 1.4$\times$ speedup over CMix-NN and MCUNet respectively under the same resource constraints.

MCU-MixQ: A HW/SW Co-optimized Mixed-precision Neural Network Design Framework for MCUs

TL;DR

This work addresses the challenge of deploying mixed-precision neural networks on resource-constrained MCUs by proposing MCU-MixQ, a HW/SW co-design framework that jointly optimizes quantization and operator implementation. It combines SIMD-based low-bitwidth operator packing (SLBC) with a differentiable NAS-based hardware-aware quantization explorer, and integrates with TinyEngine for MCU deployment. A performance predictor guides the NAS to balance accuracy and runtime efficiency, while adaptive packing and reordering further improve SIMD utilization. Empirical results show significant speedups over CMix-NN and MCUNet under the same constraints, validating the practicality of co-optimizing both quantization and operator design for MCU inference.

Abstract

Mixed-precision neural network (MPNN) that utilizes just enough data width for the neural network processing is an effective approach to meet the stringent resources constraints including memory and computing of MCUs. Nevertheless, there is still a lack of sub-byte and mixed-precision SIMD operations in MCU-class ISA and the limited computing capability of MCUs remains underutilized, which further aggravates the computing bound encountered in neural network processing. As a result, the benefits of MPNNs cannot be fully unleashed. In this work, we propose to pack multiple low-bitwidth arithmetic operations within a single instruction multiple data (SIMD) instructions in typical MCUs, and then develop an efficient convolution operator by exploring both the data parallelism and computing parallelism in convolution along with the proposed SIMD packing. Finally, we further leverage Neural Architecture Search (NAS) to build a HW/SW co-designed MPNN design framework, namely MCU-MixQ. This framework can optimize both the MPNN quantization and MPNN implementation efficiency, striking an optimized balance between neural network performance and accuracy. According to our experiment results, MCU-MixQ achieves 2.1 and 1.4 speedup over CMix-NN and MCUNet respectively under the same resource constraints.
Paper Structure (18 sections, 1 theorem, 11 equations, 8 figures, 1 table, 2 algorithms)

This paper contains 18 sections, 1 theorem, 11 equations, 8 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Mixed-Precision Nerual Network Quantization
  4. Network Operator Optimizations
  5. HW/SW Co-Optimization
  6. Overall Design Framework
  7. Low-bitwidth Network Operators
  8. Hardware-Aware MPNN Quantization
  9. Low-bitwidth Network Operator Optimizations
  10. SIMD-based Low-bitwidth Convolution
  11. Enhance Locality Through Reordering
  12. Adaptive SIMD Packing
  13. Packing Performance Prediction
  14. Experiment
  15. End-to-End Performance Evaluation
  16. ...and 3 more sections

Key Result

Theorem 4.1

For SIMD registers with $L$ lanes, each lane can accommodate $N$ low-bitwidth elements, a group of $N\times L^{2}$ elements will be reordered and packed within $L$ SIMD registers. For the $y$th lane of the $x$th SIMD register, it will be packed into the $y$th position of the $x$th one.

Figures (8)

  • Figure 1: MCU-MixQ Overview, the proposed HW/SW co-optimization design framework for MPNN on MCUs.
  • Figure 2: SIMD Low-bitwidth Convolution (SLBC) Overview
  • Figure 3: Output overlapping in naïve SLBC
  • Figure 4: Output Overlapping in Reordered SLDC
  • Figure 5: Speedup over naive and SIMD convolution
  • ...and 3 more figures

Theorems & Definitions (1)

  • Theorem 4.1