Hardware-Aware DNN Compression for Homogeneous Edge Devices

TL;DR

The paper addresses performance variability across homogeneous edge devices by introducing HDAP, a hardware-aware pruning framework that clusters devices and uses surrogate-based evaluation to minimize average inference latency under an accuracy constraint. It combines a population-based pruning (NCS) with two-stage surrogate modeling (DBSCAN clustering and GBRT predictors) to dramatically reduce real-device evaluations while maintaining performance. Empirical results on ImageNet with ResNet50 and MobileNetV1 show substantial latency reductions (up to 2.86x at 1.0G FLOPs) and robust cross-device consistency, with surrogate-guided pruning closely matching hardware-guided pruning. The approach enables scalable, high-performance DNN deployment for large homogeneous edge-device clusters, offering practical impact for AIoT at scale.

Abstract

Deploying deep neural networks (DNNs) across homogeneous edge devices (the devices with the same SKU labeled by the manufacturer) often assumes identical performance among them. However, once a device model is widely deployed, the performance of each device becomes different after a period of running. This is caused by the differences in user configurations, environmental conditions, manufacturing variances, battery degradation, etc. Existing DNN compression methods have not taken this scenario into consideration and can not guarantee good compression results in all homogeneous edge devices. To address this, we propose Homogeneous-Device Aware Pruning (HDAP), a hardware-aware DNN compression framework explicitly designed for homogeneous edge devices, aiming to achieve optimal average performance of the compressed model across all devices. To deal with the difficulty of time-consuming hardware-aware evaluations for thousands or millions of homogeneous edge devices, HDAP partitions all the devices into several device clusters, which can dramatically reduce the number of devices to evaluate and use the surrogate-based evaluation instead of hardware evaluation in real-time. Experiments on ResNet50 and MobileNetV1 with the ImageNet dataset show that HDAP consistently achieves lower average inference latency compared with state-of-the-art methods, with substantial speedup gains (e.g., 2.86 $\times$ speedup at 1.0G FLOPs for ResNet50) on the homogeneous device clusters. HDAP offers an effective solution for scalable, high-performance DNN deployment methods for homogeneous edge devices.

Figures (6)

• Figure 1: Inference latency of a DNN model on homogeneous edge clusters. (a), (c): noticeable latency variation across devices in Jetson Xavier NX (nx) and Nano clusters (nano). (b), (d): significant differences in cumulative distribution function curves of three sample devices per cluster.
• Figure 2: Different hardware evaluation methods.
• Figure 3: Overview of HDAP. It consists of iterative pruning guided by surrogate-based evaluation, and fine-tuning. Surrogates are built via clustering and supervised learning if $M$ is outside the model space.
• Figure 4: Latency distributions across device clusters for ResNet50 under three FLOPs budget. HDAP achieves the best performance, as indicated by ‘$\star$’, achieving the lowest maximum and minimum latencies across device clusters.
• Figure 5: Accuracy of surrogate models for different construction methods on four DNN models. The Clustering-based method achieves prediction accuracy close to the Per-device method while outperforming the Unified method.