Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Multi-Objective Evolutionary Neural Architecture Search for Recurrent Neural Networks

Reinhard Booysen, Anna Sergeevna Bosman

TL;DR

The results show that the proposed method is capable of finding novel RNN architectures with comparable performance to state-of-the-art manually designed RNN architectures, but with reduced computational demand.

Abstract

Artificial neural network (NN) architecture design is a nontrivial and time-consuming task that often requires a high level of human expertise. Neural architecture search (NAS) serves to automate the design of NN architectures and has proven to be successful in automatically finding NN architectures that outperform those manually designed by human experts. NN architecture performance can be quantified based on multiple objectives, which include model accuracy and some NN architecture complexity objectives, among others. The majority of modern NAS methods that consider multiple objectives for NN architecture performance evaluation are concerned with automated feed forward NN architecture design, which leaves multi-objective automated recurrent neural network (RNN) architecture design unexplored. RNNs are important for modeling sequential datasets, and prominent within the natural language processing domain. It is often the case in real world implementations of machine learning and NNs that a reasonable trade-off is accepted for marginally reduced model accuracy in favour of lower computational resources demanded by the model. This paper proposes a multi-objective evolutionary algorithm-based RNN architecture search method. The proposed method relies on approximate network morphisms for RNN architecture complexity optimisation during evolution. The results show that the proposed method is capable of finding novel RNN architectures with comparable performance to state-of-the-art manually designed RNN architectures, but with reduced computational demand.

Multi-Objective Evolutionary Neural Architecture Search for Recurrent Neural Networks

TL;DR

The results show that the proposed method is capable of finding novel RNN architectures with comparable performance to state-of-the-art manually designed RNN architectures, but with reduced computational demand.

Abstract

Artificial neural network (NN) architecture design is a nontrivial and time-consuming task that often requires a high level of human expertise. Neural architecture search (NAS) serves to automate the design of NN architectures and has proven to be successful in automatically finding NN architectures that outperform those manually designed by human experts. NN architecture performance can be quantified based on multiple objectives, which include model accuracy and some NN architecture complexity objectives, among others. The majority of modern NAS methods that consider multiple objectives for NN architecture performance evaluation are concerned with automated feed forward NN architecture design, which leaves multi-objective automated recurrent neural network (RNN) architecture design unexplored. RNNs are important for modeling sequential datasets, and prominent within the natural language processing domain. It is often the case in real world implementations of machine learning and NNs that a reasonable trade-off is accepted for marginally reduced model accuracy in favour of lower computational resources demanded by the model. This paper proposes a multi-objective evolutionary algorithm-based RNN architecture search method. The proposed method relies on approximate network morphisms for RNN architecture complexity optimisation during evolution. The results show that the proposed method is capable of finding novel RNN architectures with comparable performance to state-of-the-art manually designed RNN architectures, but with reduced computational demand.
Paper Structure (21 sections, 7 equations, 15 figures, 8 tables, 1 algorithm)

This paper contains 21 sections, 7 equations, 15 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Recurrent Neural Network Architecture
  4. Evolutionary Multi-Objective Optimisation
  5. Multi-Objective Evolutionary NAS
  6. Recurrent Neural Network NAS
  7. Multi-Objective Evolutionary algorithm for Recurrent Neural Architecture Search
  8. Search Space
  9. Encoding
  10. Search Method
  11. Approximate Recurrent Neural Network Morphism
  12. Initial Population
  13. Fitness Evaluation
  14. Selection
  15. Empirical Results
  16. ...and 6 more sections

Figures (15)

  • Figure 1: Overview of the MOE/RNAS algorithm
  • Figure 2: The rdm68_45 architecture evolved by the MOE/RNAS algorithm
  • Figure 3: Word-level NLP task average number of blocks per generation for a single run
  • Figure 4: Word-level NLP task average test perplexity per generation for a single run
  • Figure 5: The Pareto-front of the word-level NLP task experiment. RNN and LSTM architectures are included for reference
  • ...and 10 more figures