From Hand-Crafted Metrics to Evolved Training-Free Performance Predictors for Neural Architecture Search via Genetic Programming

TL;DR

The paper tackles the instability of zero-cost NAS metrics due to manual design and problem-specific biases. It introduces a Symbolic Regression via Genetic Programming framework that learns a robust, generalizable ZC proxy (SR-NAS) from existing hand-crafted metrics using a multi-problem NAS dataset and a Kendall's $ au$-based fitness. SR-NAS achieves state-of-the-art or competitive Kendall's $ au$ correlations across NAS-Bench-Suite-Zero benchmarks and generalizes to unseen spaces such as TransNAS-Bench-101-Micro/Macro, DARTS, and OFA, enabling rapid NAS iterations on modest hardware (approximately $10$–$15$ minutes). The framework is extensible, allowing inclusion of additional metrics and problems to further enhance cross-problem performance and practical applicability in real-world NAS tasks.

Abstract

Estimating the network performance using zero-cost (ZC) metrics has proven both its efficiency and efficacy in Neural Architecture Search (NAS). However, a notable limitation of most ZC proxies is their inconsistency, as reflected by the substantial variation in their performance across different problems. Furthermore, the design of existing ZC metrics is manual, involving a time-consuming trial-and-error process that requires substantial domain expertise. These challenges raise two critical questions: (1) Can we automate the design of ZC metrics? and (2) Can we utilize the existing hand-crafted ZC metrics to synthesize a more generalizable one? In this study, we propose a framework based on Symbolic Regression via Genetic Programming to automate the design of ZC metrics. Our framework is not only highly extensible but also capable of quickly producing a ZC metric with a strong positive rank correlation to true network performance across diverse NAS search spaces and tasks. Extensive experiments on 13 problems from NAS-Bench-Suite-Zero demonstrate that our automatically generated proxies consistently outperform hand-crafted alternatives. Using our evolved proxy metric as the search objective in an evolutionary algorithm, we could identify network architectures with competitive performance within 15 minutes using a single consumer GPU.

Figures (6)

• Figure 1: Comparisons in Kendall's Tau rank correlations between our automatically-designed ZC metric and state-of-the-art ZC metrics (i.e., MeCo, ZiCo, and SWAP) on NAS-Bench-101/201/301 (Left) and TransNAS-Bench-101-Micro/Macro (Right). Highest Kendall's $\tau$ scores for each problem are presented. The results highlight the consistency of our ZC metric across various search spaces and tasks.
• Figure 2: Example of using expression tree to represent the expression $\textbf{Snip} \times (\textbf{Snip} + {\textbf{MeCo}})$ in Symbolic Regression.
• Figure 3: Illustration of crossover and mutation operators for expression trees.
• Figure 4: The procedure of searching new ZC metrics with Symbolic Regression via Genetic Programming. Our dataset consists of multiple NAS problems and it thus contains multiple search spaces $SS$ = {$SS_1$, $SS_2$, $\ldots$, $SS_i$} and tasks $T$ = {$T_1$, $T_2$, $\ldots$, $T_i$}. $SS_i$ refers to the $i$-th search space in the list of search spaces $SS$ and $T_j$ refers to the $j$-th in the list of tasks $T$.
• Figure 5: Frequency of hand-crafted ZC metrics in the 31 final ZC metrics synthesized by SR.