LimitNet: Progressive, Content-Aware Image Offloading for Extremely Weak Devices & Networks

TL;DR

LimitNet tackles the challenge of running vision inference on extremely weak devices over bandwidth-constrained LPWANs by introducing a lightweight, progressive, content-aware image encoder. The edge component includes a compact encoder, a saliency detector, and a Gradual Scoring mechanism that orders latent data by importance, enabling partial data to be useful for cloud inference. The cloud reconstructs missing latent values and runs a decoder and classifier, with a prioritized retransmission strategy to ensure critical information arrives within deadlines. Across ImageNet1000, CIFAR100, and COCO, LimitNet delivers substantial bandwidth savings (e.g., 61.24%–83.68%) and accuracy gains over state-of-the-art baselines, while running on Cortex-M class MCUs with modest encoding overhead. This work advances practical, real-time vision for remote IoT deployments by harmonizing low-edge computation, progressive transmission, and robust partial-data inference.

Abstract

IoT devices have limited hardware capabilities and are often deployed in remote areas. Consequently, advanced vision models surpass such devices' processing and storage capabilities, requiring offloading of such tasks to the cloud. However, remote areas often rely on LPWANs technology with limited bandwidth, high packet loss rates, and extremely low duty cycles, which makes fast offloading for time-sensitive inference challenging. Today's approaches, which are deployable on weak devices, generate a non-progressive bit stream, and therefore, their decoding quality suffers strongly when data is only partially available on the cloud at a deadline due to limited bandwidth or packet losses. In this paper, we introduce LimitNet, a progressive, content-aware image compression model designed for extremely weak devices and networks. LimitNet's lightweight progressive encoder prioritizes critical data during transmission based on the content of the image, which gives the cloud the opportunity to run inference even with partial data availability. Experimental results demonstrate that LimitNet, on average, compared to SOTA, achieves 14.01 p.p. (percentage point) higher accuracy on ImageNet1000, 18.01 pp on CIFAR100, and 0.1 higher mAP@0.5 on COCO. Also, on average, LimitNet saves 61.24% bandwidth on ImageNet1000, 83.68% on CIFAR100, and 42.25% on the COCO dataset compared to SOTA, while it only has 4% more encoding time compared to JPEG (with a fixed quality) on STM32F7 (Cortex-M7).

Figures (13)

• Figure 1: Qualitative comparison of LimitNet's progressive reconstruction. LimitNet detects the important parts of the image and sends the encoded data in the order of importance. Progressive offloading allows the cloud to run the inference at any point (lightning symbol). The horizontal axis shows the offloading bitstream of an image, the first number shows the size of received data (in KB) at a specific time, followed by the corresponding Top5 Accuracy (Top5-Acc) of EfficientNet-B0 li2020efficient on ImageNet1000 deng2009imagenet using this received data.
• Figure 2: Overview of LimitNet. Our encoder compresses the input to a latent representation. Gradual Scoring and our saliency detector extract the important parts of the latent data and assign an importance score to each latent data point. Afterward, LimitNet starts to transmit the latent to the cloud in order of its importance score. On the cloud side, at any given time, we can reconstruct the latent by setting unreceived values to zero and feeding it to a powerful decoder. After reconstruction, we feed the output to a vision model redmon2016yololi2020efficient.
• Figure 3: The architecture of LimitNet is inspired by ResNet he2016deep and adopts larger kernels for the first layers. All the used layers are supported by the DSP accelerator of the MCUs.
• Figure 4: Reconstruction outputs and Top5-Acc (on ImageNet1000) at different data availability levels when we only use the saliency map compared to combining it with Gradual Scoring.
• Figure 5: Details of Gradual Scoring. This Figure illustrates where and how we add $G_{Factor}$ to each filter's activations of the latent, enabling the model to learn and choose how much background, i.e., context, it needs.