Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-conditioned Graph Diffusion for Neural Architecture Search

Rohan Asthana, Joschua Conrad, Youssef Dawoud, Maurits Ortmanns, Vasileios Belagiannis

TL;DR

This work presents a graph diffusion-based NAS approach that uses discrete conditional graph diffusion processes to generate high-performing neural network architectures and proposes a multi-conditioned classifier-free guidance approach applied to graph diffusion networks to jointly impose constraints such as high accuracy and low hardware latency.

Abstract

Neural architecture search automates the design of neural network architectures usually by exploring a large and thus complex architecture search space. To advance the architecture search, we present a graph diffusion-based NAS approach that uses discrete conditional graph diffusion processes to generate high-performing neural network architectures. We then propose a multi-conditioned classifier-free guidance approach applied to graph diffusion networks to jointly impose constraints such as high accuracy and low hardware latency. Unlike the related work, our method is completely differentiable and requires only a single model training. In our evaluations, we show promising results on six standard benchmarks, yielding novel and unique architectures at a fast speed, i.e. less than 0.2 seconds per architecture. Furthermore, we demonstrate the generalisability and efficiency of our method through experiments on ImageNet dataset.

Multi-conditioned Graph Diffusion for Neural Architecture Search

TL;DR

This work presents a graph diffusion-based NAS approach that uses discrete conditional graph diffusion processes to generate high-performing neural network architectures and proposes a multi-conditioned classifier-free guidance approach applied to graph diffusion networks to jointly impose constraints such as high accuracy and low hardware latency.

Abstract

Neural architecture search automates the design of neural network architectures usually by exploring a large and thus complex architecture search space. To advance the architecture search, we present a graph diffusion-based NAS approach that uses discrete conditional graph diffusion processes to generate high-performing neural network architectures. We then propose a multi-conditioned classifier-free guidance approach applied to graph diffusion networks to jointly impose constraints such as high accuracy and low hardware latency. Unlike the related work, our method is completely differentiable and requires only a single model training. In our evaluations, we show promising results on six standard benchmarks, yielding novel and unique architectures at a fast speed, i.e. less than 0.2 seconds per architecture. Furthermore, we demonstrate the generalisability and efficiency of our method through experiments on ImageNet dataset.
Paper Structure (51 sections, 15 equations, 3 figures, 11 tables, 2 algorithms)

This paper contains 51 sections, 15 equations, 3 figures, 11 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neural Architecture Search (NAS)
  4. Diffusion Models
  5. Background
  6. Denoising Diffusion Probabilistic Models
  7. Conditional diffusion models with guidance
  8. Discrete Graph Diffusion
  9. Training
  10. Sampling
  11. Method
  12. Diffusion based NAS
  13. Unconditional model
  14. Conditional model
  15. Discretisation of the target variable
  16. ...and 36 more sections

Figures (3)

  • Figure 1: Overview of our approach. We train a discrete graph diffusion model (denoted as $q_D$) on valid architectures from the architecture search space along with their performance metrics (eg. accuracy, latency). After training, we sample architectures given the required conditions (eg. accuracy>top5%, latency<2).
  • Figure 2: Examples of high performing generated cells by our method DiNAS on DARTS search space using NAS-Bench-301.
  • Figure 3: Examples of high performing generated cells by DARTS darts (left) and TENAS chen2020tenas (right)