Multi-Objective Hardware Aware Neural Architecture Search using Hardware Cost Diversity

TL;DR

This work tackles the high computational cost of hardware-aware NAS by proposing MO-HDNAS, a three-objective evolutionary method that optimizes a representation-similarity score to a reference model $\phi(\alpha^*, \alpha)$, a hardware cost $\Psi(\alpha)$, and a hardware-cost-diversity term $\chi(\alpha, \mathcal{P})$ to promote exploration. By leveraging NSGA-II, MO-HDNAS discovers a Pareto front of architectures in a single run across six edge devices, reducing search time versus prior single-objective approaches. The framework is evaluated on NAS-Bench-201 with HW-NAS-Bench data across CIFAR-10/100 and ImageNet-16-120, demonstrating diverse, high-performing architectures with substantially lower GPU-hours (e.g., $\approx$32x faster) and improved exploration due to the diversity objective. Ablation studies show that hardware-cost diversity sustains population diversity across generations, enabling the discovery of architectures with a broader range of latency-accuracy trade-offs. The proposed method offers a practical, efficient pathway for hardware-aware NAS on resource-constrained platforms.

Abstract

Hardware-aware Neural Architecture Search approaches (HW-NAS) automate the design of deep learning architectures, tailored specifically to a given target hardware platform. Yet, these techniques demand substantial computational resources, primarily due to the expensive process of assessing the performance of identified architectures. To alleviate this problem, a recent direction in the literature has employed representation similarity metric for efficiently evaluating architecture performance. Nonetheless, since it is inherently a single objective method, it requires multiple runs to identify the optimal architecture set satisfying the diverse hardware cost constraints, thereby increasing the search cost. Furthermore, simply converting the single objective into a multi-objective approach results in an under-explored architectural search space. In this study, we propose a Multi-Objective method to address the HW-NAS problem, called MO-HDNAS, to identify the trade-off set of architectures in a single run with low computational cost. This is achieved by optimizing three objectives: maximizing the representation similarity metric, minimizing hardware cost, and maximizing the hardware cost diversity. The third objective, i.e. hardware cost diversity, is used to facilitate a better exploration of the architecture search space. Experimental results demonstrate the effectiveness of our proposed method in efficiently addressing the HW-NAS problem across six edge devices for the image classification task.

Figures (6)

• Figure 1: An illustration of the difference between a single objective approach to HW-NAS problem and our proposed method MO-HDNAS.
• Figure 2: Results of a naive conversion from a single objective to a multi-objective NAS with two objectives: maximize representation similarity and minimizing device latency. It fails to identify the best architecture within the search space. The architecture search is performed in the search space, NAS-Bench-201 Dong2020NAS-Bench-201 on CIFAR10 dataset. More details about the search space are given in Section \ref{['subsection:search_space']}.
• Figure 3: A depiction of the accuracy (y-axis) of all architectures against their respective hardware costs measured in terms of latency (x-axis), in a population size of 5. As the value of diversity increases, the architectures in the population exhibit a spread in hardware costs along the latency axis.
• Figure 4: Results of MO-HDNAS$\ $ for 6 different edge devices performed with only 3 objectives: maximize representation similarity, minimizing device latency and maximizing the hardware cost diversity. (a), (b), (c) show the results for image classification task on CIFAR10, CIFAR100 and ImageNet16-120 respectively.
• Figure 5: Comparison of architecture search results for FPGA on CIFAR-100 dataset between HW-EvRSNAS sinha2024hardware and MO-HDNAS.