Efficient Few-Shot Neural Architecture Search by Counting the Number of Nonlinear Functions

TL;DR

This work tackles the inefficiency of weight-sharing in NAS by partitioning the search space into subspaces defined by the number of nonlinear functions, and assigning a dedicated, channel-reduced supernet to each subspace. A novel training scheme, supernet-balanced sampling (SBS), trains multiple subnets from different subspaces in parallel, enabling efficient multi-supernet training on a single machine. After training, an evolutionary search selects the best architecture from the combined space using subspace-specific supernet parameters. The approach achieves state-of-the-art or competitive results on NAS benchmarks with significantly reduced computational overhead compared to existing few-shot NAS methods, highlighting the practicality of nonlinear-function-based space division and SBS for scalable NAS.

Abstract

Neural architecture search (NAS) enables finding the best-performing architecture from a search space automatically. Most NAS methods exploit an over-parameterized network (i.e., a supernet) containing all possible architectures (i.e., subnets) in the search space. However, the subnets that share the same set of parameters are likely to have different characteristics, interfering with each other during training. To address this, few-shot NAS methods have been proposed that divide the space into a few subspaces and employ a separate supernet for each subspace to limit the extent of weight sharing. They achieve state-of-the-art performance, but the computational cost increases accordingly. We introduce in this paper a novel few-shot NAS method that exploits the number of nonlinear functions to split the search space. To be specific, our method divides the space such that each subspace consists of subnets with the same number of nonlinear functions. Our splitting criterion is efficient, since it does not require comparing gradients of a supernet to split the space. In addition, we have found that dividing the space allows us to reduce the channel dimensions required for each supernet, which enables training multiple supernets in an efficient manner. We also introduce a supernet-balanced sampling (SBS) technique, sampling several subnets at each training step, to train different supernets evenly within a limited number of training steps. Extensive experiments on standard NAS benchmarks demonstrate the effectiveness of our approach. Our code is available at https://cvlab.yonsei.ac.kr/projects/EFS-NAS.

Figures (5)

• Figure 1: Illustration of search space splitting strategies. Individual supernets are highlighted in different colors. Subnets with similar characteristics are marked by the same shape. Left: FS-NAS zhao2021few splits the space randomly. Although the random splitting strategy is efficient, each supernet could contain subnets that are likely to conflict with each other. Middle: GM-NAS hu2022generalizing compares gradients of a supernet to split the space, better grouping subnets. This however incurs a lot of computational cost. Right: We propose to count the number of nonlinear functions within a subnet such that each subspace contains subnets with the same number of nonlinear functions only. Our splitting criterion incurs negligible overheads, while separating the space effectively. Best viewed in color.
• Figure 2: Distributions of top-1 test accuracies for subnets of each supernet on ImageNet-16-120 chrabaszcz2017downsampled of NAS201 dong2020bench. Each dotted line represents the median value for the corresponding distribution. (a) FS-NAS zhao2021few adopts five supernets, dividing the search space randomly. (b-e) We leverage zero-cost proxies to divide the space, resulting in three supernets. They are (from left to right) FLOPs ning2021evaluatingli2023zico, the number of linear regions mellor2021neural, an isotropy of a feature space lee2024az, and the number of nonlinear functions. Best viewed in color.
• Figure 3: Histograms of two variables (i.e., FLOPs and the number of parameters) for subnets of each supernet, where a red star indicates a bin with the highest frequency. We can see that our splitting criterion makes subnets from different supernets have unique structures in terms of FLOPs and the number of parameters. This suggests that our approach enables better training subnets, since the subnets with similar structures are less likely to suffer from the conflicts, compared with the ones with different structures. Best viewed in color.
• Figure 4: Analysis of the performance of subnets with varying the number of channels for supernets. The x-axis shows accuracies of subnets sampled from supernets using full channel dimensions (i.e., $G$=1), while the y-axis represents those sampled from supernets with reduced channel (i.e., $G$=2). We measure the rank correlation in terms of Kendall's tau scores kendall1938new, particularly for high-performing subnets (i.e., top 150 subnets). (a-c) We have observed that exploiting multiple supernets enables better preserving the performance ranking. (d-e) We speculate that this is because the performance ranking among subnets from different supernets is likely to be maintained, as the subnets belonging to different supernets do not interfere with each other. Note that we highlight each supernet in a different color. Best viewed in color.
• Figure 5: Left: We visualize three supernets, where the supernet colored in orange has the largest number of subnets $N_1$. Right: Randomly sampling a single subnet from the entire space $\mathcal{A}$ (=$\mathcal{A}_0 \cup \mathcal{A}_1 \cup \mathcal{A}_2$) at each training step causes the training imbalance between supernets. That is, the training process is biased towards sampling subnets from the space of $\mathcal{A}_1$ (top). Instead, our SBS samples multiple subnets at each training step, where each subnet is sampled from a different supernet (bottom). This allows us to train supernets evenly within a finite number of training steps. Best viewed in color.