Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

An algorithmic framework for the optimization of deep neural networks architectures and hyperparameters

Julie Keisler, El-Ghazali Talbi, Sandra Claudel, Gilles Cabriel

TL;DR

DRAGON proposes a DAG-based Neural Architecture Search and Hyperparameter Optimization framework, using an asynchronous evolutionary algorithm to jointly optimize architecture and hyperparameters for time series forecasting. The flexible, adjacency-matrix DAG encoding supports mixtures of operations including self-attention, enabling non-traditional, high-performing DNNs tailored to time series data. Empirical results on the Monash benchmark show DRAGON outperforms 11 of 27 handcrafted/AutoML baselines and remains competitive with AutoGluon, while noting computation-time considerations and model simplicity. The work highlights the potential of DAG-based AutoDL for domains lacking clearly defined architectures and outlines avenues for speedups and enhancements, such as multi-fidelity search and ensemble integration.

Abstract

In this paper, we propose an algorithmic framework to automatically generate efficient deep neural networks and optimize their associated hyperparameters. The framework is based on evolving directed acyclic graphs (DAGs), defining a more flexible search space than the existing ones in the literature. It allows mixtures of different classical operations: convolutions, recurrences and dense layers, but also more newfangled operations such as self-attention. Based on this search space we propose neighbourhood and evolution search operators to optimize both the architecture and hyper-parameters of our networks. These search operators can be used with any metaheuristic capable of handling mixed search spaces. We tested our algorithmic framework with an evolutionary algorithm on a time series prediction benchmark. The results demonstrate that our framework was able to find models outperforming the established baseline on numerous datasets.

An algorithmic framework for the optimization of deep neural networks architectures and hyperparameters

TL;DR

DRAGON proposes a DAG-based Neural Architecture Search and Hyperparameter Optimization framework, using an asynchronous evolutionary algorithm to jointly optimize architecture and hyperparameters for time series forecasting. The flexible, adjacency-matrix DAG encoding supports mixtures of operations including self-attention, enabling non-traditional, high-performing DNNs tailored to time series data. Empirical results on the Monash benchmark show DRAGON outperforms 11 of 27 handcrafted/AutoML baselines and remains competitive with AutoGluon, while noting computation-time considerations and model simplicity. The work highlights the potential of DAG-based AutoDL for domains lacking clearly defined architectures and outlines avenues for speedups and enhancements, such as multi-fidelity search and ensemble integration.

Abstract

In this paper, we propose an algorithmic framework to automatically generate efficient deep neural networks and optimize their associated hyperparameters. The framework is based on evolving directed acyclic graphs (DAGs), defining a more flexible search space than the existing ones in the literature. It allows mixtures of different classical operations: convolutions, recurrences and dense layers, but also more newfangled operations such as self-attention. Based on this search space we propose neighbourhood and evolution search operators to optimize both the architecture and hyper-parameters of our networks. These search operators can be used with any metaheuristic capable of handling mixed search spaces. We tested our algorithmic framework with an evolutionary algorithm on a time series prediction benchmark. The results demonstrate that our framework was able to find models outperforming the established baseline on numerous datasets.
Paper Structure (29 sections, 5 equations, 11 figures, 6 tables)

This paper contains 29 sections, 5 equations, 11 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Deep learning for time series forecasting
  4. Search spaces for automated deep learning
  5. AutoML for time series forecasting
  6. Search space definition
  7. Optimization problem formulation
  8. Architecture Search Space
  9. Hyperparameters Search Space
  10. Search algorithm
  11. Evolutionary algorithm design
  12. Architecture evolution
  13. Mutation.
  14. Crossover.
  15. Hyperparameters evolution
  16. ...and 14 more sections

Figures (11)

  • Figure 1: Classification of encoding strategies for NAS talbi2021automated.
  • Figure 2: DNN encoding as a directed acyclic graph (DAG). The elements in blue (crosshatch) are fixed by the framework, the architecture elements from $\alpha$ are displayed in beige and the hyperparameters $\lambda$ are in pink (dots).
  • Figure 3: Evolutionary algorithm flowchart.
  • Figure 4: Crossover operator illustration.
  • Figure 5: Meta-architecture for Monash time series datasets.
  • ...and 6 more figures