Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Task-Specific Activation Functions for Neuroevolution using Grammatical Evolution

Benjamin David Winter, William John Teahan

TL;DR

This work addresses the challenge of selecting activation functions that suit a specific task by automatically evolving them with Grammatical Evolution. Using a Backus–Naur grammar, Neuvo GEAF maps genotypes to novel activation functions and integrates them into neural networks without increasing parameter counts. Across four binary classification datasets, task-specific evolved activations consistently improve F1-scores by 2.4%–9.4% over ReLU, demonstrating potential for edge-efficient, high-performance networks. The approach highlights the value of neuroevolution in designing activation functions and sets the stage for combining activation-function evolution with broader architecture search and multi-objective optimization.

Abstract

Activation functions play a critical role in the performance and behaviour of neural networks, significantly impacting their ability to learn and generalise. Traditional activation functions, such as ReLU, sigmoid, and tanh, have been widely used with considerable success. However, these functions may not always provide optimal performance for all tasks and datasets. In this paper, we introduce Neuvo GEAF - an innovative approach leveraging grammatical evolution (GE) to automatically evolve novel activation functions tailored to specific neural network architectures and datasets. Experiments conducted on well-known binary classification datasets show statistically significant improvements in F1-score (between 2.4% and 9.4%) over ReLU using identical network architectures. Notably, these performance gains were achieved without increasing the network's parameter count, supporting the trend toward more efficient neural networks that can operate effectively on resource-constrained edge devices. This paper's findings suggest that evolved activation functions can provide significant performance improvements for compact networks while maintaining energy efficiency during both training and inference phases.

Task-Specific Activation Functions for Neuroevolution using Grammatical Evolution

TL;DR

This work addresses the challenge of selecting activation functions that suit a specific task by automatically evolving them with Grammatical Evolution. Using a Backus–Naur grammar, Neuvo GEAF maps genotypes to novel activation functions and integrates them into neural networks without increasing parameter counts. Across four binary classification datasets, task-specific evolved activations consistently improve F1-scores by 2.4%–9.4% over ReLU, demonstrating potential for edge-efficient, high-performance networks. The approach highlights the value of neuroevolution in designing activation functions and sets the stage for combining activation-function evolution with broader architecture search and multi-objective optimization.

Abstract

Activation functions play a critical role in the performance and behaviour of neural networks, significantly impacting their ability to learn and generalise. Traditional activation functions, such as ReLU, sigmoid, and tanh, have been widely used with considerable success. However, these functions may not always provide optimal performance for all tasks and datasets. In this paper, we introduce Neuvo GEAF - an innovative approach leveraging grammatical evolution (GE) to automatically evolve novel activation functions tailored to specific neural network architectures and datasets. Experiments conducted on well-known binary classification datasets show statistically significant improvements in F1-score (between 2.4% and 9.4%) over ReLU using identical network architectures. Notably, these performance gains were achieved without increasing the network's parameter count, supporting the trend toward more efficient neural networks that can operate effectively on resource-constrained edge devices. This paper's findings suggest that evolved activation functions can provide significant performance improvements for compact networks while maintaining energy efficiency during both training and inference phases.

Paper Structure

This paper contains 12 sections, 3 figures, 6 tables.

Table of Contents

  1. Introduction
  2. Background and Related Work
  3. Grammatical Evolution
  4. Grammatical Neuroevolution
  5. Activation functions
  6. Methodology
  7. Genetic operators
  8. The Grammar
  9. Experimental Setup
  10. Datasets
  11. Experimental Results and Discussion
  12. Conclusion and Future Work

Figures (3)

  • Figure 1: Activation functions produced from the phenotype for the WBCD dataset. Activation functions can be found in Table \ref{['table:geaf_results']}. Results are found in Table \ref{['tab:comparative_results']}.
  • Figure 2: Activation functions produced from the phenotype for the Pima Indian Diabetes Detection dataset. Activation functions can be found in Table \ref{['table:geaf_results']}. Results are found in Table \ref{['tab:comparative_results']}.
  • Figure 3: The activation functions generated from the phenotype for the Heart Disease Detection dataset are listed in Table \ref{['table:geaf_results']}, with the corresponding results provided in Table \ref{['tab:comparative_results']}.