Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

NeuroLGP-SM: A Surrogate-assisted Neuroevolution Approach using Linear Genetic Programming

Fergal Stapleton, Brendan Cody-Kenny, Edgar Galván

TL;DR

Neuroevolution of DNNs is computationally expensive, motivating surrogate-assisted approaches. The authors introduce NeuroLGP-SM, a framework that combines Linear Genetic Programming (NeuroLGP) with Kriging Partial Least Squares (KPLS) and a repair-based baseline to manage high-dimensional phenotypes via phenotypic distance, guided by an EI acquisition strategy. Across 96 runs on BreakHis, NeuroLGP-SM consistently matches or outperforms the baseline while reducing GPU-hour requirements by approximately 25%, with the baseline itself providing competitive benchmarks relative to VGG-16. The work demonstrates robust performance across multiple magnifications and highlights a practical path to efficient, topology-aware neuroevolution for image classification. Overall, the approach offers scalable, surrogate-assisted neuroevolution suitable for complex DNN searches with reduced computational demands.

Abstract

Evolutionary algorithms are increasingly recognised as a viable computational approach for the automated optimisation of deep neural networks (DNNs) within artificial intelligence. This method extends to the training of DNNs, an approach known as neuroevolution. However, neuroevolution is an inherently resource-intensive process, with certain studies reporting the consumption of thousands of GPU days for refining and training a single DNN network. To address the computational challenges associated with neuroevolution while still attaining good DNN accuracy, surrogate models emerge as a pragmatic solution. Despite their potential, the integration of surrogate models into neuroevolution is still in its early stages, hindered by factors such as the effective use of high-dimensional data and the representation employed in neuroevolution. In this context, we address these challenges by employing a suitable representation based on Linear Genetic Programming, denoted as NeuroLGP, and leveraging Kriging Partial Least Squares. The amalgamation of these two techniques culminates in our proposed methodology known as the NeuroLGP-Surrogate Model (NeuroLGP-SM). For comparison purposes, we also code and use a baseline approach incorporating a repair mechanism, a common practice in neuroevolution. Notably, the baseline approach surpasses the renowned VGG-16 model in accuracy. Given the computational intensity inherent in DNN operations, a singular run is typically the norm. To evaluate the efficacy of our proposed approach, we conducted 96 independent runs. Significantly, our methodologies consistently outperform the baseline, with the SM model demonstrating superior accuracy or comparable results to the NeuroLGP approach. Noteworthy is the additional advantage that the SM approach exhibits a 25% reduction in computational requirements, further emphasising its efficiency for neuroevolution.

NeuroLGP-SM: A Surrogate-assisted Neuroevolution Approach using Linear Genetic Programming

TL;DR

Neuroevolution of DNNs is computationally expensive, motivating surrogate-assisted approaches. The authors introduce NeuroLGP-SM, a framework that combines Linear Genetic Programming (NeuroLGP) with Kriging Partial Least Squares (KPLS) and a repair-based baseline to manage high-dimensional phenotypes via phenotypic distance, guided by an EI acquisition strategy. Across 96 runs on BreakHis, NeuroLGP-SM consistently matches or outperforms the baseline while reducing GPU-hour requirements by approximately 25%, with the baseline itself providing competitive benchmarks relative to VGG-16. The work demonstrates robust performance across multiple magnifications and highlights a practical path to efficient, topology-aware neuroevolution for image classification. Overall, the approach offers scalable, surrogate-assisted neuroevolution suitable for complex DNN searches with reduced computational demands.

Abstract

Evolutionary algorithms are increasingly recognised as a viable computational approach for the automated optimisation of deep neural networks (DNNs) within artificial intelligence. This method extends to the training of DNNs, an approach known as neuroevolution. However, neuroevolution is an inherently resource-intensive process, with certain studies reporting the consumption of thousands of GPU days for refining and training a single DNN network. To address the computational challenges associated with neuroevolution while still attaining good DNN accuracy, surrogate models emerge as a pragmatic solution. Despite their potential, the integration of surrogate models into neuroevolution is still in its early stages, hindered by factors such as the effective use of high-dimensional data and the representation employed in neuroevolution. In this context, we address these challenges by employing a suitable representation based on Linear Genetic Programming, denoted as NeuroLGP, and leveraging Kriging Partial Least Squares. The amalgamation of these two techniques culminates in our proposed methodology known as the NeuroLGP-Surrogate Model (NeuroLGP-SM). For comparison purposes, we also code and use a baseline approach incorporating a repair mechanism, a common practice in neuroevolution. Notably, the baseline approach surpasses the renowned VGG-16 model in accuracy. Given the computational intensity inherent in DNN operations, a singular run is typically the norm. To evaluate the efficacy of our proposed approach, we conducted 96 independent runs. Significantly, our methodologies consistently outperform the baseline, with the SM model demonstrating superior accuracy or comparable results to the NeuroLGP approach. Noteworthy is the additional advantage that the SM approach exhibits a 25% reduction in computational requirements, further emphasising its efficiency for neuroevolution.
Paper Structure (16 sections, 3 equations, 4 figures, 1 table)

This paper contains 16 sections, 3 equations, 4 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Neuroevolution using Surrogate Models
  4. Background
  5. Surrogate assisted models: Kriging and Kriging Partial Least Squares
  6. Phenotypic Distance
  7. Methodology
  8. NeuroLGP
  9. NeuroLGP with Surrogate Model (NeuroLGP-SM)
  10. Genetic Operations and Repair Mechanism
  11. Experimental Setup
  12. Results
  13. Preliminary Analysis of the Baseline Model
  14. Comparison of Models
  15. Analysis of the Surrogate Model
  16. ...and 1 more sections

Figures (4)

  • Figure 1: left: NeuroLGP psuedocode for python. right: Functional API example in TensorFlow.
  • Figure 2: Left: Diagram showing the interplay between a typical evolutionary algorithm and a surrogate model approach. Right: The surrogate model management strategy is shown on a more granular level.
  • Figure 3: Accuracy of 6000 individuals (read text).
  • Figure 4: Figures \ref{['fig:violin40']} through \ref{['fig:violin400']} plot violin plots, showing the distribution of accuracies of the architectures with the best fitness for each of the four data sets across 8 runs.