A Survey on Neural Architecture Search

TL;DR

NAS seeks to automatically discover high-performing architectures by casting architecture design as a black-box optimization over defined search spaces. The survey formalizes global, cell-based, and recurrent search spaces and analyzes optimization methods including reinforcement learning, evolutionary algorithms, surrogate-model-based optimization, and one-shot approaches, highlighting trade-offs, scalability, and transferability. It further covers extensions such as early termination, transfer learning, multi-objective optimization, and model compression, and discusses critical challenges, baselines, and future directions toward sample-efficient, holistic automation of the deep learning pipeline. Overall, the NAS landscape remains diverse, with NASNet-inspired spaces dominating, weight-sharing and differentiable approaches driving speed, and open questions about true novelty versus search-space biases guiding future research.

Abstract

The growing interest in both the automation of machine learning and deep learning has inevitably led to the development of a wide variety of automated methods for neural architecture search. The choice of the network architecture has proven to be critical, and many advances in deep learning spring from its immediate improvements. However, deep learning techniques are computationally intensive and their application requires a high level of domain knowledge. Therefore, even partial automation of this process helps to make deep learning more accessible to both researchers and practitioners. With this survey, we provide a formalism which unifies and categorizes the landscape of existing methods along with a detailed analysis that compares and contrasts the different approaches. We achieve this via a comprehensive discussion of the commonly adopted architecture search spaces and architecture optimization algorithms based on principles of reinforcement learning and evolutionary algorithms along with approaches that incorporate surrogate and one-shot models. Additionally, we address the new research directions which include constrained and multi-objective architecture search as well as automated data augmentation, optimizer and activation function search.

Figures (19)

• Figure 1: Global search spaces: (a) chain-structured, (b) with skips, (c) architecture template, only the connections between the dark blue (dashed) operations are not fixed.
• Figure 2: Tan2018_MnasNet propose to decompose the architecture into different segments. Each segment $i$ has its own pattern (blue operations) which is repeated $n_i$ times and has $f_i$ filters.
• Figure 3: Structure of the NASNet search space instances. $n$ normal cells followed by a reduction cell. This sequence is repeated several times, the reduction cell might be repeated. This decision is a hyperparameter and depends on the image resolution. The 1x1* convolution is a special operation which converts $\boldsymbol{z}^{\left(n\right)}$ to match the shape of $\boldsymbol{z}^{\left(n+1\right)}$.
• Figure 4: Reduction cell of the NASNet-A architecture Zoph2018_Learning as one example how a cell in the NASNet search space can look like. Blocks can be used as input for other blocks (e.g. block 1 and 3), unused blocks are concatenated and are the output of the cell.
• Figure 5: Architecture template (left) and the Block-QNN-B cell discovered by Zhong2018_Practical (right).