Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TG-NAS: Generalizable Zero-Cost Proxies with Operator Description Embedding and Graph Learning for Efficient Neural Architecture Search

Ye Qiao, Jingcheng Li, Haocheng Xu, Sitao Huang

TL;DR

TG-NAS introduces a universal zero-cost NAS proxy that remains effective without retraining by marrying a Transformer-based operator embedding with a Graph Convolutional Network predictor. This combination enables cross-space generalization to unseen operators and search spaces, delivering strong ranking fidelity and dramatic search-time reductions. Across 12 NAS benchmarks, TG-NAS achieves state-of-the-art or competitive correlation with ground-truth performance, while delivering up to 300x speedups over prior zero-cost methods and competitive ImageNet results in the DARTS space. The work advances NAS by providing a data-efficient, space-agnostic proxy that can serve as a foundational tool for fast, generalizable architecture search.

Abstract

Neural Architecture Search (NAS) is a powerful technique for discovering high-performing CNN architectures, but most existing methods rely on costly training or extensive sampling. Zero-shot NAS offers a training-free alternative by using proxies to predict architecture performance. However, existing proxies are often suboptimal -- frequently outperformed by simple metrics like parameter count or FLOPs -- and they generalize poorly across different search spaces. Moreover, current model-based proxies struggle to adapt to new operators without access to ground-truth accuracy, limiting their transferability. We propose TG-NAS, a universal, model-based zero-cost (ZC) proxy that combines a Transformer-based operator embedding generator with a Graph Convolutional Network (GCN) to predict architecture performance. Unlike prior model-based predictors, TG-NAS requires no retraining and generalizes across arbitrary search spaces. It serves as a standalone ZC proxy with strong data efficiency, robustness, and cross-space consistency. Extensive evaluations across diverse NAS benchmarks demonstrate TG-NAS's superior rank correlation and generalizability compared to existing proxies. Additionally, it improves search efficiency by up to 300x and discovers architectures achieving 93.75% CIFAR-10 accuracy on NAS-Bench-201 and 74.9% ImageNet top-1 accuracy on the DARTS space, establishing TG-NAS as a promising foundation for efficient, generalizable NAS.

TG-NAS: Generalizable Zero-Cost Proxies with Operator Description Embedding and Graph Learning for Efficient Neural Architecture Search

TL;DR

TG-NAS introduces a universal zero-cost NAS proxy that remains effective without retraining by marrying a Transformer-based operator embedding with a Graph Convolutional Network predictor. This combination enables cross-space generalization to unseen operators and search spaces, delivering strong ranking fidelity and dramatic search-time reductions. Across 12 NAS benchmarks, TG-NAS achieves state-of-the-art or competitive correlation with ground-truth performance, while delivering up to 300x speedups over prior zero-cost methods and competitive ImageNet results in the DARTS space. The work advances NAS by providing a data-efficient, space-agnostic proxy that can serve as a foundational tool for fast, generalizable architecture search.

Abstract

Neural Architecture Search (NAS) is a powerful technique for discovering high-performing CNN architectures, but most existing methods rely on costly training or extensive sampling. Zero-shot NAS offers a training-free alternative by using proxies to predict architecture performance. However, existing proxies are often suboptimal -- frequently outperformed by simple metrics like parameter count or FLOPs -- and they generalize poorly across different search spaces. Moreover, current model-based proxies struggle to adapt to new operators without access to ground-truth accuracy, limiting their transferability. We propose TG-NAS, a universal, model-based zero-cost (ZC) proxy that combines a Transformer-based operator embedding generator with a Graph Convolutional Network (GCN) to predict architecture performance. Unlike prior model-based predictors, TG-NAS requires no retraining and generalizes across arbitrary search spaces. It serves as a standalone ZC proxy with strong data efficiency, robustness, and cross-space consistency. Extensive evaluations across diverse NAS benchmarks demonstrate TG-NAS's superior rank correlation and generalizability compared to existing proxies. Additionally, it improves search efficiency by up to 300x and discovers architectures achieving 93.75% CIFAR-10 accuracy on NAS-Bench-201 and 74.9% ImageNet top-1 accuracy on the DARTS space, establishing TG-NAS as a promising foundation for efficient, generalizable NAS.
Paper Structure (24 sections, 2 equations, 11 figures, 8 tables, 1 algorithm)

This paper contains 24 sections, 2 equations, 11 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Neural Architecture Search (NAS)
  4. Zero-Cost (ZC) Proxies for NAS
  5. Challenges in Model-based Prediction
  6. TG-NAS: A Model-based Zero-cost Proxy
  7. Architecture Representation
  8. Operator Embedding Generator
  9. GCN Proxy Predictor
  10. Evolutionary Searching
  11. Experiment and Results Analysis
  12. Generalizability Analysis of ZC Proxies Across 12 Benchmarks
  13. Independent Analysis of proxies
  14. NAS Result on NAS-Bench-201
  15. NAS Result for ImageNet on the DARTS Search Space
  16. ...and 9 more sections

Figures (11)

  • Figure 1: Spearman rank correlation comparison between ours and 4 other ZC proxies (the higher the better).
  • Figure 2: TG-NAS, featuring operator description embedding and GCN-based performance predictor
  • Figure 3: Spearman's $\rho$ rank between ZC proxy values and ground truth accuracies, for 14 ZC proxies and across 12 NAS benchmarks (the higher the better).
  • Figure 4: Spearman's $\rho$ Correlation between ZC proxy scores of CIFAR-10 on NAS-Bench-201.
  • Figure 5: TG-NAS vs. Ground Truth ranking on NAS-Bench-201 Space
  • ...and 6 more figures