NeuroLGP-SM: Scalable Surrogate-Assisted Neuroevolution for Deep Neural Networks

TL;DR

The proposed approach, named Neuro-Linear Genetic Programming surrogate model (NeuroLGP-SM), efficiently and accurately estimates DNN fitness without the need for complete evaluations, demonstrates competitive or superior results compared to 12 other methods, including NeuroLGP without SM, convolutional neural networks, support vector machines, and autoencoders.

Abstract

Evolutionary Algorithms (EAs) play a crucial role in the architectural configuration and training of Artificial Deep Neural Networks (DNNs), a process known as neuroevolution. However, neuroevolution is hindered by its inherent computational expense, requiring multiple generations, a large population, and numerous epochs. The most computationally intensive aspect lies in evaluating the fitness function of a single candidate solution. To address this challenge, we employ Surrogate-assisted EAs (SAEAs). While a few SAEAs approaches have been proposed in neuroevolution, none have been applied to truly large DNNs due to issues like intractable information usage. In this work, drawing inspiration from Genetic Programming semantics, we use phenotypic distance vectors, outputted from DNNs, alongside Kriging Partial Least Squares (KPLS), an approach that is effective in handling these large vectors, making them suitable for search. Our proposed approach, named Neuro-Linear Genetic Programming surrogate model (NeuroLGP-SM), efficiently and accurately estimates DNN fitness without the need for complete evaluations. NeuroLGP-SM demonstrates competitive or superior results compared to 12 other methods, including NeuroLGP without SM, convolutional neural networks, support vector machines, and autoencoders. Additionally, it is worth noting that NeuroLGP-SM is 25% more energy-efficient than its NeuroLGP counterpart. This efficiency advantage adds to the overall appeal of our proposed NeuroLGP-SM in optimising the configuration of large DNNs.

Figures (6)

• Figure 1: left: LGP example in C language. Based on example from brameier2007linear. right: NeuroLGP psuedocode for python.
• Figure 2: Diagram of the NeuroLGP genotype-to-phenotype mapping. The pseudocode for the set of instructions (left-hand side) can be represented as the genotype with effective and non-effective code (top) and produces the resulting phenotype (bottom right-hand side) as a specific neural network architecture. Note that the non-effective coding is not present in the phenotype.
• Figure 3: Model management strategy in terms of fitness evaluation, model management and training the surrogate model.
• Figure 4: Avg. MSE between predicted vs. actual fitness over 15 generations for the NeuroLGP-SM approach. The relative stability shows the robustness of the NeuroLGP-SM approach.
• Figure 5: Predicted vs. actual accuracies (black points), for $\times$40, $\times$100, $\times$200, and $\times$400 datasets, shown in (a) -- (d), respectively, across 8 independent runs for the surrogate-assisted approach (NeuroLGP-SM). The red line denotes where the accuracy for both predicted and actual are the same, where points closer to this line are preferential.