Efficient Evaluation Methods for Neural Architecture Search: A Survey

TL;DR

A comprehensively survey of EEMs published up to date, and divides the existing EEMs into four categories based on the number of DNNs trained for constructing these EEMs, thus making easily understanding the research trends of EEMs.

Abstract

Neural Architecture Search (NAS) has received increasing attention because of its exceptional merits in automating the design of Deep Neural Network (DNN) architectures. However, the performance evaluation process, as a key part of NAS, often requires training a large number of DNNs. This inevitably makes NAS computationally expensive. In past years, many Efficient Evaluation Methods (EEMs) have been proposed to address this critical issue. In this paper, we comprehensively survey these EEMs published up to date, and provide a detailed analysis to motivate the further development of this research direction. Specifically, we divide the existing EEMs into four categories based on the number of DNNs trained for constructing these EEMs. The categorization can reflect the degree of efficiency in principle, which can in turn help quickly grasp the methodological features. In surveying each category, we further discuss the design principles and analyze the strengths and weaknesses to clarify the landscape of existing EEMs, thus making easily understanding the research trends of EEMs. Furthermore, we also discuss the current challenges and issues to identify future research directions in this emerging topic. In summary, this survey provides a convenient overview of EEM for interested users, and they can easily select the proper EEM method for the tasks at hand. In addition, the researchers in the NAS field could continue exploring the future directions suggested in the paper.

Figures (13)

• Figure 1: The number of "submissions" refers to EEMs. The statistical results are searched on Google Scholar with the following steps: (1) Select a specific year; (2) Search with the keywords "early stopping" OR "learning curve" OR "network morphism" OR "weight sharing" OR "one-shot" OR "DARTS" OR "differentiable" OR "predictor" OR "population memory" OR "zero-cost" OR "zero-shot" OR "training-free" AND "architecture search" OR "architecture design" OR "CNN" OR "deep learning" OR "deep neural network"; (3) Check the selected papers in detail to verify if they belong to EEMs.
• Figure 2: The workflow of the NAS algorithm.
• Figure 3: The flowchart of $N$-shot evaluation method and its relationship to subcategory methods.
• Figure 4: The organization of the description for EEMs.
• Figure 5: An example of width morphing, where different types of nodes indicate distinct operations (e.g., convolution layer, pooling layer). The labels on the edges represent the corresponding weight values. The parent network is a fully-trained network, and the child network is morphed from the parent network by a width morphing operation. Specifically, in the child network, node #6 is a copy of node #3, and the weights of node #3 are also transferred to node #6. To preserve the function of the two networks, the value of weight $b$ is divided by two, while the values of weights $a$, $c$, $d$, and $e$ remain unchanged. In this way, if the same input is fed to the parent network and the child network, respectively, the output of both networks will be the same, which means the function of the parent network is preserved in the child network.