Efficient Global Neural Architecture Search
Shahid Siddiqui, Christos Kyrkou, Theocharis Theocharides
TL;DR
Efficient Global Neural Architecture Search tackles NAS’s prohibitive compute by introducing a navigable macro–micro search space and architecture-aware ranking that uses dynamic training protocols to fairly compare diverse networks. The method decouples macro depth/width discovery from micro per-layer refinements, enabling efficient exploration under tight parameter budgets, with a bilevel optimization formulation guiding search. Across CIFAR-10/100, FashionMNIST, EMNIST, KMNIST, and face-recognition benchmarks, the approach yields competitive or state-of-the-art accuracy with 2–4x faster search than leading global NAS methods and demonstrates transferability to real-world face recognition tasks. The framework automates end-to-end network discovery, producing compact, high-performing models suitable for practical deployment, and offers a foundation for joint macro-architecture and training-protocol optimization in future work.
Abstract
Neural architecture search (NAS) has shown promise towards automating neural network design for a given task, but it is computationally demanding due to training costs associated with evaluating a large number of architectures to find the optimal one. To speed up NAS, recent works limit the search to network building blocks (modular search) instead of searching the entire architecture (global search), approximate candidates' performance evaluation in lieu of complete training, and use gradient descent rather than naturally suitable discrete optimization approaches. However, modular search does not determine network's macro architecture i.e. depth and width, demanding manual trial and error post-search, hence lacking automation. In this work, we revisit NAS and design a navigable, yet architecturally diverse, macro-micro search space. In addition, to determine relative rankings of candidates, existing methods employ consistent approximations across entire search spaces, whereas different networks may not be fairly comparable under one training protocol. Hence, we propose an architecture-aware approximation with variable training schemes for different networks. Moreover, we develop an efficient search strategy by disjoining macro-micro network design that yields competitive architectures in terms of both accuracy and size. Our proposed framework achieves a new state-of-the-art on EMNIST and KMNIST, while being highly competitive on the CIFAR-10, CIFAR-100, and FashionMNIST datasets and being 2-4x faster than the fastest global search methods. Lastly, we demonstrate the transferability of our framework to real-world computer vision problems by discovering competitive architectures for face recognition applications.