Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Delta-NAS: Difference of Architecture Encoding for Predictor-based Evolutionary Neural Architecture Search

Arjun Sridhar, Yiran Chen

TL;DR

This work proposes projecting the problem to a lower dimensional space through pre-dicting the difference in accuracy of a pair of similar net-works to allow for reducing computational complexity from exponential down to linear with re-spect to the size of the search space.

Abstract

Neural Architecture Search (NAS) continues to serve a key roll in the design and development of neural networks for task specific deployment. Modern NAS techniques struggle to deal with ever increasing search space complexity and compute cost constraints. Existing approaches can be categorized into two buckets: fine-grained computational expensive NAS and coarse-grained low cost NAS. Our objective is to craft an algorithm with the capability to perform fine-grain NAS at a low cost. We propose projecting the problem to a lower dimensional space through predicting the difference in accuracy of a pair of similar networks. This paradigm shift allows for reducing computational complexity from exponential down to linear with respect to the size of the search space. We present a strong mathematical foundation for our algorithm in addition to extensive experimental results across a host of common NAS Benchmarks. Our methods significantly out performs existing works achieving better performance coupled with a significantly higher sample efficiency.

Delta-NAS: Difference of Architecture Encoding for Predictor-based Evolutionary Neural Architecture Search

TL;DR

This work proposes projecting the problem to a lower dimensional space through pre-dicting the difference in accuracy of a pair of similar net-works to allow for reducing computational complexity from exponential down to linear with re-spect to the size of the search space.

Abstract

Neural Architecture Search (NAS) continues to serve a key roll in the design and development of neural networks for task specific deployment. Modern NAS techniques struggle to deal with ever increasing search space complexity and compute cost constraints. Existing approaches can be categorized into two buckets: fine-grained computational expensive NAS and coarse-grained low cost NAS. Our objective is to craft an algorithm with the capability to perform fine-grain NAS at a low cost. We propose projecting the problem to a lower dimensional space through predicting the difference in accuracy of a pair of similar networks. This paradigm shift allows for reducing computational complexity from exponential down to linear with respect to the size of the search space. We present a strong mathematical foundation for our algorithm in addition to extensive experimental results across a host of common NAS Benchmarks. Our methods significantly out performs existing works achieving better performance coupled with a significantly higher sample efficiency.

Paper Structure

This paper contains 20 sections, 6 theorems, 1 equation, 5 figures, 1 table, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evaluation Strategy: Predictor-based NAS
  4. Search Strategy: Evolutionary Methods
  5. Network Encodings
  6. Search Space
  7. Zero-Cost NAS
  8. Theoretical Analysis
  9. Methods
  10. Difference of Architectures
  11. NAS algorithm
  12. Experimental Results
  13. Setup
  14. Datasets and Search Spaces
  15. Implementation Details
  16. ...and 5 more sections

Key Result

Theorem 1

Given a block-based search space represented by a list of operations $\vert \mathcal{A} \vert = r^n$

Figures (5)

  • Figure 1: Through taking the difference of architectures that are similar, we are able to project to a sparse representation. We are able to scale to large search spaces linearly.
  • Figure 2: The left side of the figure shows the process for generating the difference of architecture dataset. Both pairs of graph and operation embeddings along with both accuracies are stored to create the DoA dataset. The right side of the figure shows the DoA predictor used to generate neighbors during the modified evolutionary search.
  • Figure 3: Predictor loss increases greatly as number of edits increases beyond 1 and plateaus.
  • Figure 4: Delta-NAS converges significantly faster than existing encoding schemes and evolutionary based methods.
  • Figure 5: In larger search spaces, the difference in performance between Delta-NAS and previous works is more pronounced.

Theorems & Definitions (10)

  • Definition 1
  • Theorem 1
  • Theorem 2
  • Theorem 3
  • proof
  • Theorem 4
  • proof
  • Corollary 4.1
  • Theorem 5
  • proof