Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

ECToNAS: Evolutionary Cross-Topology Neural Architecture Search

Elisabeth J. Schiessler, Roland C. Aydin, Christian J. Cyron

TL;DR

ECToNAS tackles the problem of finding suitable neural architectures under tight computational budgets by introducing a cross-topology, evolutionary NAS method that reuses weights during mutation and integrates training into the search. The approach uses a two-phase, greediness-controlled fitness function to balance accuracy gains against model size, enabling both topology growth and reduction across CNN and FFNN forms. Empirical results on six datasets show improved test accuracy over baselines and substantial compression without retraining from scratch, highlighting the method's practical efficiency for users with limited resources. The work demonstrates topology crossing in a resource-friendly framework and provides a starting point for broader cross-topology NAS research with potential extensions like dynamic budgets and early stopping.

Abstract

We present ECToNAS, a cost-efficient evolutionary cross-topology neural architecture search algorithm that does not require any pre-trained meta controllers. Our framework is able to select suitable network architectures for different tasks and hyperparameter settings, independently performing cross-topology optimisation where required. It is a hybrid approach that fuses training and topology optimisation together into one lightweight, resource-friendly process. We demonstrate the validity and power of this approach with six standard data sets (CIFAR-10, CIFAR-100, EuroSAT, Fashion MNIST, MNIST, SVHN), showcasing the algorithm's ability to not only optimise the topology within an architectural type, but also to dynamically add and remove convolutional cells when and where required, thus crossing boundaries between different network types. This enables researchers without a background in machine learning to make use of appropriate model types and topologies and to apply machine learning methods in their domains, with a computationally cheap, easy-to-use cross-topology neural architecture search framework that fully encapsulates the topology optimisation within the training process.

ECToNAS: Evolutionary Cross-Topology Neural Architecture Search

TL;DR

ECToNAS tackles the problem of finding suitable neural architectures under tight computational budgets by introducing a cross-topology, evolutionary NAS method that reuses weights during mutation and integrates training into the search. The approach uses a two-phase, greediness-controlled fitness function to balance accuracy gains against model size, enabling both topology growth and reduction across CNN and FFNN forms. Empirical results on six datasets show improved test accuracy over baselines and substantial compression without retraining from scratch, highlighting the method's practical efficiency for users with limited resources. The work demonstrates topology crossing in a resource-friendly framework and provides a starting point for broader cross-topology NAS research with potential extensions like dynamic budgets and early stopping.

Abstract

We present ECToNAS, a cost-efficient evolutionary cross-topology neural architecture search algorithm that does not require any pre-trained meta controllers. Our framework is able to select suitable network architectures for different tasks and hyperparameter settings, independently performing cross-topology optimisation where required. It is a hybrid approach that fuses training and topology optimisation together into one lightweight, resource-friendly process. We demonstrate the validity and power of this approach with six standard data sets (CIFAR-10, CIFAR-100, EuroSAT, Fashion MNIST, MNIST, SVHN), showcasing the algorithm's ability to not only optimise the topology within an architectural type, but also to dynamically add and remove convolutional cells when and where required, thus crossing boundaries between different network types. This enables researchers without a background in machine learning to make use of appropriate model types and topologies and to apply machine learning methods in their domains, with a computationally cheap, easy-to-use cross-topology neural architecture search framework that fully encapsulates the topology optimisation within the training process.
Paper Structure (20 sections, 6 equations, 7 figures, 6 tables, 1 algorithm)

This paper contains 20 sections, 6 equations, 7 figures, 6 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Works
  3. Methods
  4. Genetic Neural Architecture Search
  5. Available Network Operations
  6. Architecture Evaluation
  7. Experiments
  8. Convolutions on Tabular Data Set
  9. Conclusion
  10. Further Details on Network Operations
  11. Fully Connected Layers
  12. Pooling Layers
  13. Pooling Layer Trials
  14. Additional Results
  15. Implementation Details
  16. ...and 5 more sections

Figures (7)

  • Figure 1: Graphical representation of the ECToNAS algorithm
  • Figure 2: Bracket style competitions. Top: In phase 1, one winning candidate is selected per mutation type. Offspring are randomly paired into groups of two out of their assigned pools, lucky tickets occur in case of an odd number of competitors. Bottom left: In phase 2, $n$ overall winners are selected from phase 1 winners. Here, $n=2$. These will form the next parent generation.
  • Figure 3: Example of a three layer CNN. Removing a channel from $L_1$ (coloured purple) affects all incoming and outgoing connections of that layer. Created using LeNail2019
  • Figure 4: Left: Outgoing weights from the last convolutional layer are re-ordered by the flatten layer. Only one channel is depicted. Right: We mimic max pooling with stride 2, kernel size 2 on the outgoing weights of the last convolutional layer. White blocks are discarded.
  • Figure 5: Comparison of accuracy on the validation set achieved by ECToNAS (greedy mode, $\alpha = 1$) with random mode and unmodified baseline topology across different data sets. Graphs are aggregated over all starting topology variations.
  • ...and 2 more figures