SONATA: Self-adaptive Evolutionary Framework for Hardware-aware Neural Architecture Search

TL;DR

This work proposes SONATA, a self-adaptive evolutionary algorithm for HW-aware NAS that leverages adaptive evolutionary operators guided by the learned importance of NN design parameters through tree-based surrogate models and a Reinforcement Learning agent.

Abstract

Recent advancements in Artificial Intelligence (AI), driven by Neural Networks (NN), demand innovative neural architecture designs, particularly within the constrained environments of Internet of Things (IoT) systems, to balance performance and efficiency. HW-aware Neural Architecture Search (HW-aware NAS) emerges as an attractive strategy to automate the design of NN using multi-objective optimization approaches, such as evolutionary algorithms. However, the intricate relationship between NN design parameters and HW-aware NAS optimization objectives remains an underexplored research area, overlooking opportunities to effectively leverage this knowledge to guide the search process accordingly. Furthermore, the large amount of evaluation data produced during the search holds untapped potential for refining the optimization strategy and improving the approximation of the Pareto front. Addressing these issues, we propose SONATA, a self-adaptive evolutionary algorithm for HW-aware NAS. Our method leverages adaptive evolutionary operators guided by the learned importance of NN design parameters. Specifically, through tree-based surrogate models and a Reinforcement Learning agent, we aspire to gather knowledge on 'How' and 'When' to evolve NN architectures. Comprehensive evaluations across various NAS search spaces and hardware devices on the ImageNet-1k dataset have shown the merit of SONATA with up to 0.25% improvement in accuracy and up to 2.42x gains in latency and energy. Our SONATA has seen up to sim$93.6% Pareto dominance over the native NSGA-II, further stipulating the importance of self-adaptive evolution operators in HW-aware NAS.

Figures (9)

• Figure 1: The workflow of a typical Evolutionary NAS framework (ENAS).
• Figure 2: NSGA-II selection and evolution mechanisms. $P_T$ corresponds to current population at iteration T, $Q_T$ is the offspring population at iteration T. $PF_{(1:N)}$ are the Pareto fronts retrieved by the non-dominates sorting algorithm -- such that genomes in $PF_i$ dominate the genomes in $PF_{i+1}$.
• Figure 3: Calculation of the crowding distance. To simplify the interpretability, we give an example of two optimization objectives, $y^1$ and $y^2$. The same technique can be applied to the case of more than two objectives.
• Figure 4: An overview of our novel HW-aware NAS framework: SONATA self-adaptive and data-driven evolutionary search process.
• Figure 5: SONATA Neural network (NN) encoding scheme.