Pareto-wise Ranking Classifier for Multi-objective Evolutionary Neural Architecture Search

TL;DR

A classifier-based Pareto evolution approach is proposed, where an online classifier is trained to directly predict the dominance relationship between the candidate and reference architectures and is able to alleviate the rank disorder issue and outperforms other methods.

Abstract

In the deployment of deep neural models, how to effectively and automatically find feasible deep models under diverse design objectives is fundamental. Most existing neural architecture search (NAS) methods utilize surrogates to predict the detailed performance (e.g., accuracy and model size) of a candidate architecture during the search, which however is complicated and inefficient. In contrast, we aim to learn an efficient Pareto classifier to simplify the search process of NAS by transforming the complex multi-objective NAS task into a simple Pareto-dominance classification task. To this end, we propose a classification-wise Pareto evolution approach for one-shot NAS, where an online classifier is trained to predict the dominance relationship between the candidate and constructed reference architectures, instead of using surrogates to fit the objective functions. The main contribution of this study is to change supernet adaption into a Pareto classifier. Besides, we design two adaptive schemes to select the reference set of architectures for constructing classification boundary and regulate the rate of positive samples over negative ones, respectively. We compare the proposed evolution approach with state-of-the-art approaches on widely-used benchmark datasets, and experimental results indicate that the proposed approach outperforms other approaches and have found a number of neural architectures with different model sizes ranging from 2M to 6M under diverse objectives and constraints.

Figures (8)

• Figure 1: An illustration of the framework of CENAS, where the numbers indicate the sequence of the operations.
• Figure 2: An illustration of the search space of CENAS. Top: the outer structure (omitting skip inputs for clarity) for architectural search. Bottom: an example to show the densely-connection via the skip operations when $K$ is set to three, where the normal cell group consists of three normal cells.
• Figure 3: Left: An example of a pairwise combination and corresponding encoding. Top: an example to illustrate the structure of a cell represented by a directed acyclic graph with seven nodes highlighted in green (nodes 0,$\ldots$,6). Bottom: an example of encoding matrix to represent the above cell.
• Figure 4: An example to show the impact of different distributions of reference points (denoted by red stars) to the classification. The boundary of plot (a) is formed by six randomly selected reference points, while the boundary of plot (b) is formed by six different reference points from two segments of the front.
• Figure 5: An example to show the difference of the number of dominated solutions by using the $\alpha$-domination with different $\alpha$. (a) a larger $\alpha$ leads to a larger dominated region of a solution and smaller number of positive samples will be obtained. (b) a smaller $\alpha$ leads to smaller dominated region and more positive samples.