Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Training Feedforward Neural Networks with Bayesian Hyper-Heuristics

Arné Schreuder, Anna Bosman, Andries Engelbrecht, Christopher Cleghorn

TL;DR

This research introduces a novel population-based Bayesian hyper-heuristic (BHH) that is used to train feedforward neural networks (FFNNs) and provides an automated method for finding the best heuristic to train the FFNNs at various stages of the training process.

Abstract

The process of training feedforward neural networks (FFNNs) can benefit from an automated process where the best heuristic to train the network is sought out automatically by means of a high-level probabilistic-based heuristic. This research introduces a novel population-based Bayesian hyper-heuristic (BHH) that is used to train feedforward neural networks (FFNNs). The performance of the BHH is compared to that of ten popular low-level heuristics, each with different search behaviours. The chosen heuristic pool consists of classic gradient-based heuristics as well as meta-heuristics (MHs). The empirical process is executed on fourteen datasets consisting of classification and regression problems with varying characteristics. The BHH is shown to be able to train FFNNs well and provide an automated method for finding the best heuristic to train the FFNNs at various stages of the training process.

Training Feedforward Neural Networks with Bayesian Hyper-Heuristics

TL;DR

This research introduces a novel population-based Bayesian hyper-heuristic (BHH) that is used to train feedforward neural networks (FFNNs) and provides an automated method for finding the best heuristic to train the FFNNs at various stages of the training process.

Abstract

The process of training feedforward neural networks (FFNNs) can benefit from an automated process where the best heuristic to train the network is sought out automatically by means of a high-level probabilistic-based heuristic. This research introduces a novel population-based Bayesian hyper-heuristic (BHH) that is used to train feedforward neural networks (FFNNs). The performance of the BHH is compared to that of ten popular low-level heuristics, each with different search behaviours. The chosen heuristic pool consists of classic gradient-based heuristics as well as meta-heuristics (MHs). The empirical process is executed on fourteen datasets consisting of classification and regression problems with varying characteristics. The BHH is shown to be able to train FFNNs well and provide an automated method for finding the best heuristic to train the FFNNs at various stages of the training process.
Paper Structure (30 sections, 10 equations, 4 figures, 8 tables, 1 algorithm)

This paper contains 30 sections, 10 equations, 4 figures, 8 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Artifical Neural Networks
  3. Heuristics
  4. Hyper-Heuristics
  5. Probability
  6. Bayesian Hyper-Heuristics
  7. Heuristic Pool
  8. Proxies
  9. Entity Pool
  10. Performance Log
  11. Credit Assignment Strategy
  12. Selection Mechanism
  13. Optimisation Step
  14. Maximum A Posteriori Estimation
  15. Hyper-Parameters
  16. ...and 15 more sections

Figures (4)

  • Figure 1: An illustration of the architecture and high level components of the BHH.
  • Figure 2: Mapping of proxied heuristic state update operations as implemented by the BHH
  • Figure 3: Descriptive plots for the average ranks of all low-level heuristics compared to three heuristic pool variants of the BHH baseline configuration, per dataset, across all independent runs and epochs.
  • Figure 4: Critical difference plots for the average ranks of all low-level heuristics compared to three heuristic pool variants of the baseline BHH, across all datasets, runs and epochs.