A First Step Towards Runtime Analysis of Evolutionary Neural Architecture Search

TL;DR

This work initiates a mathematical runtime analysis for Evolutionary Neural Architecture Search (ENAS) by introducing Uniform, an explicit binary classification proxy that maps neural-architecture configurations to accuracy via a tractable fitness function. It analyzes a (1+1)-EA style ENAS with local and global mutations, proving both achieve linear expected runtime $Θ(n)$ to locate the optimum, and shows empirical equivalence between the mutation operators. The approach leverages fitness-level methods and additive drift to obtain upper and lower bounds, providing a foundational step toward rigorous ENAS theory. The results offer insight into how mutation strategies influence search efficiency and lay groundwork for extending analysis to more realistic NAS problems and operators.

Abstract

Evolutionary neural architecture search (ENAS) employs evolutionary algorithms to find high-performing neural architectures automatically, and has achieved great success. However, compared to the empirical success, its rigorous theoretical analysis has yet to be touched. This work goes preliminary steps toward the mathematical runtime analysis of ENAS. In particular, we define a binary classification problem $\textsc{UNIFORM}$, and formulate an explicit fitness function to represent the relationship between neural architecture and classification accuracy. Furthermore, we consider (1+1)-ENAS algorithm with mutation to optimize the neural architecture, and obtain the following runtime bounds: both the local and global mutations find the optimum in an expected runtime of $Θ(n)$, where $n$ is the problem size. The theoretical results show that the local and global mutations achieve nearly the same performance on $\textsc{UNIFORM}$. Empirical results also verify the equivalence of these two mutation operators.

Figures (4)

• Figure 1: Three types of blocks that can be used to build DNNs with different topological architectures. All of the neurons output Boolean value. The latter two blocks are connected with ADD neurons and can function as non-linear classifiers.
• Figure 2: (a) Illustration of Uniform ($n=16$) when $D=2$. The green regions constitute the positive points classified as 1, while the white regions constitute the negative points classified as 0. (b) One optimal solution with a fitness of 1. The blocks at optimal positions (with the optimal weights) are colored as follows: blue for A-type blocks, yellow for B-type blocks, and red for C-type blocks. The arrow points to the positively classified half-space.
• Figure 3: (a) The neural architecture $A_\mathrm{NA}$ comprises an A-type, a B-type, and a C-type. (b) The best positions of the hyperplanes in $A_\mathrm{NA}$ for solving Uniform ($n=16$), i.e., the sketch of the classifier $A_\mathrm{NN}$. Given a point (0.5,0.6) as input, the classifier $A_\mathrm{NN}$ outputs 1.
• Figure 4: Average number of generations of the (1+1)-ENAS algorithm using local and global mutations for solving UNIFORM.

Theorems & Definitions (14)

• Definition 1: Uniform
• Definition 2: Neurons Parameter Training Strategy
• Lemma 1
• Theorem 1
• proof : Proof of Theorem \ref{['theorem:1+1_onebit_upper']}
• Theorem 2
• Lemma 2
• proof
• proof : Proof of Theorem \ref{['theorem:1+1_onebit_lower']}
• Theorem 3