Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Encodings for Prediction-based Neural Architecture Search

Yash Akhauri, Mohamed S. Abdelfattah

TL;DR

This paper categorize and investigate neural encodings from three main types: structural, learned, and score-based, and extends these encodings and introduces \textit{unified encodings}, that extend NAS predictors to multiple search spaces.

Abstract

Predictor-based methods have substantially enhanced Neural Architecture Search (NAS) optimization. The efficacy of these predictors is largely influenced by the method of encoding neural network architectures. While traditional encodings used an adjacency matrix describing the graph structure of a neural network, novel encodings embrace a variety of approaches from unsupervised pretraining of latent representations to vectors of zero-cost proxies. In this paper, we categorize and investigate neural encodings from three main types: structural, learned, and score-based. Furthermore, we extend these encodings and introduce \textit{unified encodings}, that extend NAS predictors to multiple search spaces. Our analysis draws from experiments conducted on over 1.5 million neural network architectures on NAS spaces such as NASBench-101 (NB101), NB201, NB301, Network Design Spaces (NDS), and TransNASBench-101. Building on our study, we present our predictor \textbf{FLAN}: \textbf{Fl}ow \textbf{A}ttention for \textbf{N}AS. FLAN integrates critical insights on predictor design, transfer learning, and \textit{unified encodings} to enable more than an order of magnitude cost reduction for training NAS accuracy predictors. Our implementation and encodings for all neural networks are open-sourced at \href{https://github.com/abdelfattah-lab/flan_nas}{https://github.com/abdelfattah-lab/flan\_nas}.

Encodings for Prediction-based Neural Architecture Search

TL;DR

This paper categorize and investigate neural encodings from three main types: structural, learned, and score-based, and extends these encodings and introduces \textit{unified encodings}, that extend NAS predictors to multiple search spaces.

Abstract

Predictor-based methods have substantially enhanced Neural Architecture Search (NAS) optimization. The efficacy of these predictors is largely influenced by the method of encoding neural network architectures. While traditional encodings used an adjacency matrix describing the graph structure of a neural network, novel encodings embrace a variety of approaches from unsupervised pretraining of latent representations to vectors of zero-cost proxies. In this paper, we categorize and investigate neural encodings from three main types: structural, learned, and score-based. Furthermore, we extend these encodings and introduce \textit{unified encodings}, that extend NAS predictors to multiple search spaces. Our analysis draws from experiments conducted on over 1.5 million neural network architectures on NAS spaces such as NASBench-101 (NB101), NB201, NB301, Network Design Spaces (NDS), and TransNASBench-101. Building on our study, we present our predictor \textbf{FLAN}: \textbf{Fl}ow \textbf{A}ttention for \textbf{N}AS. FLAN integrates critical insights on predictor design, transfer learning, and \textit{unified encodings} to enable more than an order of magnitude cost reduction for training NAS accuracy predictors. Our implementation and encodings for all neural networks are open-sourced at \href{https://github.com/abdelfattah-lab/flan_nas}{https://github.com/abdelfattah-lab/flan\_nas}.
Paper Structure (20 sections, 5 equations, 17 figures, 14 tables)

This paper contains 20 sections, 5 equations, 17 figures, 14 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Encodings
  4. FLAN: Flow Attention Networks For NAS
  5. GNN Architecture
  6. Operation Embeddings
  7. FLAN Encodings
  8. Experiments
  9. Conclusion
  10. Impact Statement
  11. Appendix
  12. Best practices for NAS
  13. Neural Architecture Design Spaces
  14. Additional Results
  15. Neural Architecture Search on NASBench-201 CIFAR-100
  16. ...and 5 more sections

Figures (17)

  • Figure 1: The basic structure of an accuracy predictor highlights that many different types of encodings can be fed to the same prediction head to perform accuracy prediction.
  • Figure 2: Illustration of important encoding methods that are discussed and evaluated in our work.
  • Figure 3: The FLAN predictor architecture showing dual graph flow mechanisms, independent updates of operation embeddings, and the capability to concatenate supplementary encodings.
  • Figure 4: FLAN sample efficiency compared to prior work. Experimental settings match tagates and Table \ref{['tab:motivate_arch']}.
  • Figure 5: End-to-end NAS using an iterative sampling algorithm. FLAN$^{T}$ improves performance for low sample counts.
  • ...and 12 more figures