A Converting Autoencoder Toward Low-latency and Energy-efficient DNN Inference at the Edge

TL;DR

This work tackles high latency and high energy consumption of DNN inference on edge devices. It introduces CBNet, a converting autoencoder-based framework that pre-processes hard-to-classify images into easy variants of the same class, followed by inference with a lightweight, truncated BranchyNet-style DNN. The approach yields up to 4.8x latency reduction and up to 79% energy savings across datasets and devices while preserving or improving accuracy, outperforming relevant baselines such as LeNet, BranchyNet, AdaDeep, and SubFlow in edge scenarios. By eliminating cloud-offloading and leveraging an efficient on-device pipeline, CBNet offers practical improvements for real-time, energy-constrained edge AI deployments.

Abstract

Reducing inference time and energy usage while maintaining prediction accuracy has become a significant concern for deep neural networks (DNN) inference on resource-constrained edge devices. To address this problem, we propose a novel approach based on "converting" autoencoder and lightweight DNNs. This improves upon recent work such as early-exiting framework and DNN partitioning. Early-exiting frameworks spend different amounts of computation power for different input data depending upon their complexity. However, they can be inefficient in real-world scenarios that deal with many hard image samples. On the other hand, DNN partitioning algorithms that utilize the computation power of both the cloud and edge devices can be affected by network delays and intermittent connections between the cloud and the edge. We present CBNet, a low-latency and energy-efficient DNN inference framework tailored for edge devices. It utilizes a "converting" autoencoder to efficiently transform hard images into easy ones, which are subsequently processed by a lightweight DNN for inference. To the best of our knowledge, such autoencoder has not been proposed earlier. Our experimental results using three popular image-classification datasets on a Raspberry Pi 4, a Google Cloud instance, and an instance with Nvidia Tesla K80 GPU show that CBNet achieves up to 4.8x speedup in inference latency and 79% reduction in energy usage compared to competing techniques while maintaining similar or higher accuracy.

Figures (7)

• Figure 1: System Overview:- A "converting" autoencoder transforms hard images into easy images of the same class. The resulting images are fed to a lightweight model extracted from an early-exit DNN for inference.
• Figure 2: BranchyNet's inference speedup over LeNet diminishes when a significant fraction of the dataset contains hard images. Inference time was measured on a Raspberry Pi 4.
• Figure 3: Training "converting" autoencoder for image transformation.
• Figure 4: Comparing CBNet with baseline, BranchyNet, AdaDeepAdadeep, and Subflowsubflow in terms of inference latency and accuracy for MNIST dataset on Raspberry Pi 4.
• Figure 5: Scalability analysis with MNIST dataset on each hardware platform.