Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PRIOT: Pruning-Based Integer-Only Transfer Learning for Embedded Systems

Honoka Anada, Sefutsu Ryu, Masayuki Usui, Tatsuya Kaneko, Shinya Takamaeda-Yamazaki

TL;DR

The paper tackles on-device transfer learning for ultra-low-power edge devices that lack FPUs, focusing on integer-only training with static scale factors to reduce computational and memory costs. It introduces PRIOT, a pruning-based method that freezes pre-trained weights and learns a pruning pattern via edge-popup, enabling accurate transfer learning under fixed quantization scales; it also presents PRIOT-S, a memory-efficient variant that scores only a subset of edges. Empirical results on a Raspberry Pi Pico show substantial accuracy gains (8.08 to 33.75 percentage points on rotated MNIST) over static-scale baselines and meaningful memory/time trade-offs for PRIOT-S, demonstrating practical viability for embedded systems. The work offers a path to robust, on-device adaptation of neural networks on resource-constrained hardware where dynamic scaling is impractical.

Abstract

On-device transfer learning is crucial for adapting a common backbone model to the unique environment of each edge device. Tiny microcontrollers, such as the Raspberry Pi Pico, are key targets for on-device learning but often lack floating-point units, necessitating integer-only training. Dynamic computation of quantization scale factors, which is adopted in former studies, incurs high computational costs. Therefore, this study focuses on integer-only training with static scale factors, which is challenging with existing training methods. We propose a new training method named PRIOT, which optimizes the network by pruning selected edges rather than updating weights, allowing effective training with static scale factors. The pruning pattern is determined by the edge-popup algorithm, which trains a parameter named score assigned to each edge instead of the original parameters and prunes the edges with low scores before inference. Additionally, we introduce a memory-efficient variant, PRIOT-S, which only assigns scores to a small fraction of edges. We implement PRIOT and PRIOT-S on the Raspberry Pi Pico and evaluate their accuracy and computational costs using a tiny CNN model on the rotated MNIST dataset and the VGG11 model on the rotated CIFAR-10 dataset. Our results demonstrate that PRIOT improves accuracy by 8.08 to 33.75 percentage points over existing methods, while PRIOT-S reduces memory footprint with minimal accuracy loss.

PRIOT: Pruning-Based Integer-Only Transfer Learning for Embedded Systems

TL;DR

The paper tackles on-device transfer learning for ultra-low-power edge devices that lack FPUs, focusing on integer-only training with static scale factors to reduce computational and memory costs. It introduces PRIOT, a pruning-based method that freezes pre-trained weights and learns a pruning pattern via edge-popup, enabling accurate transfer learning under fixed quantization scales; it also presents PRIOT-S, a memory-efficient variant that scores only a subset of edges. Empirical results on a Raspberry Pi Pico show substantial accuracy gains (8.08 to 33.75 percentage points on rotated MNIST) over static-scale baselines and meaningful memory/time trade-offs for PRIOT-S, demonstrating practical viability for embedded systems. The work offers a path to robust, on-device adaptation of neural networks on resource-constrained hardware where dynamic scaling is impractical.

Abstract

On-device transfer learning is crucial for adapting a common backbone model to the unique environment of each edge device. Tiny microcontrollers, such as the Raspberry Pi Pico, are key targets for on-device learning but often lack floating-point units, necessitating integer-only training. Dynamic computation of quantization scale factors, which is adopted in former studies, incurs high computational costs. Therefore, this study focuses on integer-only training with static scale factors, which is challenging with existing training methods. We propose a new training method named PRIOT, which optimizes the network by pruning selected edges rather than updating weights, allowing effective training with static scale factors. The pruning pattern is determined by the edge-popup algorithm, which trains a parameter named score assigned to each edge instead of the original parameters and prunes the edges with low scores before inference. Additionally, we introduce a memory-efficient variant, PRIOT-S, which only assigns scores to a small fraction of edges. We implement PRIOT and PRIOT-S on the Raspberry Pi Pico and evaluate their accuracy and computational costs using a tiny CNN model on the rotated MNIST dataset and the VGG11 model on the rotated CIFAR-10 dataset. Our results demonstrate that PRIOT improves accuracy by 8.08 to 33.75 percentage points over existing methods, while PRIOT-S reduces memory footprint with minimal accuracy loss.

Paper Structure

This paper contains 11 sections, 3 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Background
  3. Related Work
  4. Challenges in Integer-Only Training
  5. PRIOT: Pruning-Based Integer-Only Transfer Learning
  6. Design of PRIOT
  7. PRIOT-S: Memory-Efficient Variant of PRIOT
  8. Evaluation
  9. Setup
  10. Results and Discussion
  11. Conclusion

Figures (3)

  • Figure 1: Overview of PRIOT. PRIOT trains the model by pruning the edges rather than updating the weights.
  • Figure 2: Transition of the output values of the model during the epoch when the accuracy drop happens in the existing integer-only training algorithm (NITI) with static scale factors. While the number of overflowed values ($\ge 127$) is almost zero at first (1), the output expands in the middle, and all values overflow by the end (2).
  • Figure 3: Accuracy history of each method with the rotated MNIST dataset with 30° rotation. While the accuracy of static-scale NITI drops in the middle, the accuracies of PRIOT and PRIOT-S continue to improve until the end.