Neural Architecture Search: Insights from 1000 Papers

TL;DR

This paper surveys neural architecture search (NAS) as automated neural design, tracing its rapid growth since 2020 and organizing advances into search spaces, optimization strategies, speedups, extensions, applications, benchmarks, best practices, and resources. It emphasizes the shift from traditional black-box search to one-shot and differentiable NAS (notably DARTS), weight-sharing, and hypernetworks, while detailing the trade-offs, biases, and memory considerations of each approach. Key contributions include a taxonomy of search spaces (macro, chain-structured, cell-based, hierarchical), a critical review of black-box and one-shot methods, and a comprehensive discussion of speedup techniques (learning curves, zero-cost proxies, multi-fidelity, meta-learning, weight inheritance). The survey also covers practical aspects such as benchmarks, reproducibility, and guidelines for fair comparisons, and it outlines future directions toward robust, hardware-aware, and fully automated deep learning systems. Overall, the paper provides a high-resolution map of NAS methods, benchmarks, and best practices to guide researchers and practitioners in developing scalable, automated neural architectures.

Abstract

In the past decade, advances in deep learning have resulted in breakthroughs in a variety of areas, including computer vision, natural language understanding, speech recognition, and reinforcement learning. Specialized, high-performing neural architectures are crucial to the success of deep learning in these areas. Neural architecture search (NAS), the process of automating the design of neural architectures for a given task, is an inevitable next step in automating machine learning and has already outpaced the best human-designed architectures on many tasks. In the past few years, research in NAS has been progressing rapidly, with over 1000 papers released since 2020 (Deng and Lindauer, 2021). In this survey, we provide an organized and comprehensive guide to neural architecture search. We give a taxonomy of search spaces, algorithms, and speedup techniques, and we discuss resources such as benchmarks, best practices, other surveys, and open-source libraries.

Figures (11)

• Figure 1: Number of NAS papers by year deng-21.
• Figure 2: Overview of neural architecture search nas-surveyweng2020neural. A search strategy iteratively selects architectures (typically by using an architecture encoding method) from a predefined search space $\mathcal{A}{}$. The architectures are passed to a performance estimation strategy, which returns the performance estimate to the search strategy. For one-shot methods, the search strategy and performance estimation strategy are inherently coupled.
• Figure 3: Illustration of cell-based search spaces. The outer skeleton across cells (left) is fixed, while the cells are searchable. NASNet assigns operations to nodes (middle) while DARTS assigns operations to edges (right).
• Figure 4: Illustration of hierarchical representation proposed in liu2018hierarchical. Level 1 of the hierarchy consists of choices of operation primitives. Level 2 consists of selecting the topology across small sets of operation primitives. Level 3 consists of selecting the topology across the constructions from level 2.
• Figure 5: A supernet comprises all possible architectures in the search space. Each architecture is a subnetwork (subgraph) in the supernet.