Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

HiRISE: High-Resolution Image Scaling for Edge ML via In-Sensor Compression and Selective ROI

Brendan Reidy, Sepehr Tabrizchi, Mohamadreza Mohammadi, Shaahin Angizi, Arman Roohi, Ramtin Zand

TL;DR

HiRISE tackles the challenge of deploying high-resolution vision on memory-limited edge devices by using a two-stage, ROI-driven pipeline with in-sensor grayscale conversion and pooling. The first stage runs an object-ROI detector on a compressed, analog image to locate regions of interest, while the second stage extracts the ROI from the full-resolution sensor data and digitizes only that portion for downstream tasks. The approach yields substantial reductions in data transfer, peak memory, and energy, with up to 17.7× reductions demonstrated across datasets and hardware; accuracy remains competitive, and end-to-end experiments on RAF-DB show significant practical gains. This work enables high-resolution, ROI-focused edge ML by moving substantial computation and data movement into the sensor and leveraging selective ROI to constrain processing.

Abstract

With the rise of tiny IoT devices powered by machine learning (ML), many researchers have directed their focus toward compressing models to fit on tiny edge devices. Recent works have achieved remarkable success in compressing ML models for object detection and image classification on microcontrollers with small memory, e.g., 512kB SRAM. However, there remain many challenges prohibiting the deployment of ML systems that require high-resolution images. Due to fundamental limits in memory capacity for tiny IoT devices, it may be physically impossible to store large images without external hardware. To this end, we propose a high-resolution image scaling system for edge ML, called HiRISE, which is equipped with selective region-of-interest (ROI) capability leveraging analog in-sensor image scaling. Our methodology not only significantly reduces the peak memory requirements, but also achieves up to 17.7x reduction in data transfer and energy consumption.

HiRISE: High-Resolution Image Scaling for Edge ML via In-Sensor Compression and Selective ROI

TL;DR

HiRISE tackles the challenge of deploying high-resolution vision on memory-limited edge devices by using a two-stage, ROI-driven pipeline with in-sensor grayscale conversion and pooling. The first stage runs an object-ROI detector on a compressed, analog image to locate regions of interest, while the second stage extracts the ROI from the full-resolution sensor data and digitizes only that portion for downstream tasks. The approach yields substantial reductions in data transfer, peak memory, and energy, with up to 17.7× reductions demonstrated across datasets and hardware; accuracy remains competitive, and end-to-end experiments on RAF-DB show significant practical gains. This work enables high-resolution, ROI-focused edge ML by moving substantial computation and data movement into the sensor and leveraging selective ROI to constrain processing.

Abstract

With the rise of tiny IoT devices powered by machine learning (ML), many researchers have directed their focus toward compressing models to fit on tiny edge devices. Recent works have achieved remarkable success in compressing ML models for object detection and image classification on microcontrollers with small memory, e.g., 512kB SRAM. However, there remain many challenges prohibiting the deployment of ML systems that require high-resolution images. Due to fundamental limits in memory capacity for tiny IoT devices, it may be physically impossible to store large images without external hardware. To this end, we propose a high-resolution image scaling system for edge ML, called HiRISE, which is equipped with selective region-of-interest (ROI) capability leveraging analog in-sensor image scaling. Our methodology not only significantly reduces the peak memory requirements, but also achieves up to 17.7x reduction in data transfer and energy consumption.
Paper Structure (10 sections, 4 equations, 8 figures, 3 tables)

This paper contains 10 sections, 4 equations, 8 figures, 3 tables.

Table of Contents

  1. Introduction
  2. End-to-End System Architecture
  3. In-Sensor Compression Circuit
  4. Results and Discussions
  5. Accuracy
  6. Memory Utilization
  7. Data Transfer
  8. Energy Consumption
  9. End-to-End System
  10. Conclusion

Figures (8)

  • Figure 1: Example objects of interest with (a) ROI of a 320$\times$240 image compressed in processor, (b) ROI of a 2560$\times$1920 image compressed in sensor to 320$\times$240 using HiRISE system.
  • Figure 2: Project Overview. Our proposed system aims to address three main challenges that make the use of very high-resolution cameras at conventional edge computing systems unfeasible. These challenges include energy-hungry analog-to-digital conversion, substantial data transfer overheads, and intensive memory demands. Our solution includes the implementation of a two-stage object detection model along with an analog in-sensor compression unit.
  • Figure 3: The overall architecture of the HiRISE end-to-end system including various building blocks and the data transfers between different blocks.
  • Figure 4: Analog in-sensor pooling circuit diagram.
  • Figure 5: SPICE simulation of analog compression circuits.
  • ...and 3 more figures