Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

TART: Token-based Architecture Transformer for Neural Network Performance Prediction

Yannis Y. He

TL;DR

The Token-based Architecture Transformer (TART), which predicts neural network performance without the need to train candidate networks and attains state-of-the-art performance on the DeepNets-1M dataset for performance prediction tasks without edge information, indicating the potential of Transformers to aid in discovering novel and high-performing neural architectures.

Abstract

In the realm of neural architecture design, achieving high performance is largely reliant on the manual expertise of researchers. Despite the emergence of Neural Architecture Search (NAS) as a promising technique for automating this process, current NAS methods still require human input to expand the search space and cannot generate new architectures. This paper explores the potential of Transformers in comprehending neural architectures and their performance, with the objective of establishing the foundation for utilizing Transformers to generate novel networks. We propose the Token-based Architecture Transformer (TART), which predicts neural network performance without the need to train candidate networks. TART attains state-of-the-art performance on the DeepNets-1M dataset for performance prediction tasks without edge information, indicating the potential of Transformers to aid in discovering novel and high-performing neural architectures.

TART: Token-based Architecture Transformer for Neural Network Performance Prediction

TL;DR

The Token-based Architecture Transformer (TART), which predicts neural network performance without the need to train candidate networks and attains state-of-the-art performance on the DeepNets-1M dataset for performance prediction tasks without edge information, indicating the potential of Transformers to aid in discovering novel and high-performing neural architectures.

Abstract

In the realm of neural architecture design, achieving high performance is largely reliant on the manual expertise of researchers. Despite the emergence of Neural Architecture Search (NAS) as a promising technique for automating this process, current NAS methods still require human input to expand the search space and cannot generate new architectures. This paper explores the potential of Transformers in comprehending neural architectures and their performance, with the objective of establishing the foundation for utilizing Transformers to generate novel networks. We propose the Token-based Architecture Transformer (TART), which predicts neural network performance without the need to train candidate networks. TART attains state-of-the-art performance on the DeepNets-1M dataset for performance prediction tasks without edge information, indicating the potential of Transformers to aid in discovering novel and high-performing neural architectures.
Paper Structure (16 sections, 1 equation, 6 figures, 1 table)

This paper contains 16 sections, 1 equation, 6 figures, 1 table.

Table of Contents

  1. Introduction
  2. Motivations
  3. Contributions
  4. Background
  5. Neural Architecture Search
  6. Parameter and Performance Prediction
  7. Graph Representation of Neural Architectures
  8. Token Representation of Graphs
  9. Transformer-based Generative Model
  10. Method
  11. Experiments and Results
  12. Datasets: DeepNets-1M
  13. Experiment Design
  14. Experiment 1: Pure-Transformer Predictor
  15. Experiment 2: Token-based Transformer Predictor
  16. ...and 1 more sections

Figures (6)

  • Figure 1: Current Neural Architecture Search process.
  • Figure 2: TART is an end-to-end neural predictor, which has three basic stages: 1) tokenization stage, 2) transformer learning stage, and 3) prediction stage.
  • Figure 3: Examples of computational graphs (visualized using NetworkX NetworkX). In the visualized graphs, a node is one of the 15 primitives coded with markers shown at the bottom, where they are sorted by the frequency in the training set.
  • Figure 4: Measuring correlation between the predicted and ground-truth performance of models on CIFAR-10
  • Figure 5: Although we had to halt training due to limited computational resources, our analysis of the linear regression of performance growth suggests that the predictor had not yet overfit the data.
  • ...and 1 more figures