AZ-NAS: Assembling Zero-Cost Proxies for Network Architecture Search

TL;DR

AZ-NAS is proposed, a novel approach that leverages the ensemble of various zero-cost proxies to enhance the correlation between a predicted ranking of networks and the ground truth substantially in terms of the performance, and introduces four novel zero-cost proxies that are complementary to each other.

Abstract

Training-free network architecture search (NAS) aims to discover high-performing networks with zero-cost proxies, capturing network characteristics related to the final performance. However, network rankings estimated by previous training-free NAS methods have shown weak correlations with the performance. To address this issue, we propose AZ-NAS, a novel approach that leverages the ensemble of various zero-cost proxies to enhance the correlation between a predicted ranking of networks and the ground truth substantially in terms of the performance. To achieve this, we introduce four novel zero-cost proxies that are complementary to each other, analyzing distinct traits of architectures in the views of expressivity, progressivity, trainability, and complexity. The proxy scores can be obtained simultaneously within a single forward and backward pass, making an overall NAS process highly efficient. In order to integrate the rankings predicted by our proxies effectively, we introduce a non-linear ranking aggregation method that highlights the networks highly-ranked consistently across all the proxies. Experimental results conclusively demonstrate the efficacy and efficiency of AZ-NAS, outperforming state-of-the-art methods on standard benchmarks, all while maintaining a reasonable runtime cost.

Figures (9)

• Figure 1: Comparison of training-free NAS methods on ImageNet16-120 of NAS-Bench-201 dong2020nasbench201. We compare correlation coefficients (Kendall's $\tau$) between predicted rankings of networks and the ground truth in the x-axis, and test accuracies for the selected networks in the y-axis. The runtime costs are visualized by the circle size and color. By assembling the proposed zero-cost proxies, AZ-NAS achieves the best consistency between predicted rankings of the networks and the ground truth efficiently, which helps to find the network with the highest accuracy.
• Figure 2: Toy examples for the expressivity score $s^{\mathcal{E}}$. In (a) and (b), we synthesize 2-dimensional features (green dots) using different covariances, and compare the L1-normalized coefficients of PCs. The features in (b) exhibit a higher expressivity score, forming an isotropic feature space.
• Figure 3: Correlation analysis on the zero-cost proxies of AZ-NAS. We report Kendall's $\tau$ between the estimated network rankings on ImageNet16-120 of NAS-Bench-201 dong2020nasbench201.
• Figure 4: Visual comparison of training-free NAS methods in terms of predicted network ranking ($x$-axis) vs. ground truth ($y$-axis) on ImageNet16-120 of NAS-Bench-201 dong2020nasbench201. We report the correlation coefficients between them in terms of Kendall's $\tau$ and Spearman's $\rho$, denoted by $\tau$ and $\rho$, respectively.
• Figure 5: Visual comparison of the zero-cost proxies of AZ-NAS ((a)-(d)), and the linear and non-linear ranking aggregation methods ((e) and (f)), in terms of predicted network ranking ($x$-axis) vs. ground truth ($y$-axis) on ImageNet16-120 of NAS-Bench-201 dong2020nasbench201. The colors of the points, ranging from light-yellow to dark-blue, correspond to the network ranking in (f) predicted by the AZ-NAS score. (Best viewed in color.)