Surprisingly Strong Performance Prediction with Neural Graph Features

TL;DR

The paper addresses NAS performance prediction by identifying biases and limitations in zero-cost proxies and proposing neural graph features (GRAF) as a fast, interpretable graph-based descriptor of architectures. GRAF encodes operation counts, path lengths, and node degrees from architecture graphs and, when used with a simple random forest, often outperforms zero-cost proxies and common encodings across accuracy, hardware, and robustness tasks; combining GRAF with zero-cost proxies yields the best results in many settings. The work provides extensive ablations, feature-importance analyses, and redundancy studies, showing that different tasks favor different architectural properties and that GRAF offers a practical, scalable baseline with broad applicability in NAS. Overall, GRAF acts as a strong, lightweight predictor that complements existing methods and can guide more efficient NAS, while highlighting avenues for designing task-tailored, graph-based proxies for future research.

Abstract

Performance prediction has been a key part of the neural architecture search (NAS) process, allowing to speed up NAS algorithms by avoiding resource-consuming network training. Although many performance predictors correlate well with ground truth performance, they require training data in the form of trained networks. Recently, zero-cost proxies have been proposed as an efficient method to estimate network performance without any training. However, they are still poorly understood, exhibit biases with network properties, and their performance is limited. Inspired by the drawbacks of zero-cost proxies, we propose neural graph features (GRAF), simple to compute properties of architectural graphs. GRAF offers fast and interpretable performance prediction while outperforming zero-cost proxies and other common encodings. In combination with other zero-cost proxies, GRAF outperforms most existing performance predictors at a fraction of the cost.

Figures (31)

• Figure 1: ZCP score of all networks from NB201 against the validation accuracy - colors indicate the number of conv3x3 and 1x1
• Figure 2: Spearman correlation of zero-cost proxies with validation accuracy by clusters of architectures with the same number of 1x1 and 3x3 convolutions (a,b), and only 3x3 convolutions (c). 0-6 represents clusters with corresponding convolution counts, all data is the correlation over all available networks (same as in NB-Suite-Zero nbsuitezero).
• Figure 3: Exemplary overview of neural graph features in the NAS-Bench-201 search space.
• Figure 4: Validation accuracy prediction (cifar-10) across 3 benchmarks with 1024 sampled networks. Comparison of 3 encodings (ZCP, onehot, and GRAF) and their combinations. Blue boxes denote runs including GRAF.
• Figure 5: TNB101_micro autoencoder task across 3 train sample sizes. Comparison of 3 encodings (ZCP, onehot, and GRAF) and their combinations. Blue boxes denote runs including GRAF.