A Neural Architecture Search Method using Auxiliary Evaluation Metric based on ResNet Architecture

TL;DR

The paper tackles neural architecture search (NAS) by introducing MO-ResNet, a MOEA/D-based NAS framework that optimizes both accuracy and an auxiliary validation loss $L_{val}$ within a ResNet-inspired, variable-length search space. It uses Chebyshev scalarization $g^{te}$ with weight vectors $oldsymbol{ ho}$, maintains a Pareto front, and evolves architectures via SBX crossover and polynomial mutation, training networks for two objective horizons ${nep}_{train}$ and ${nep}_{full}$. Key contributions include a novel genetic-operator suite, an extended search space built on EvoCNN augmented with skip connections, and a systematic comparison of multi-objective versus single-objective NAS. Empirical results on MNIST, Fashion-MNIST, and CIFAR-100 show competitive performance and reveal that incorporating the auxiliary metric can improve parameter-accuracy trade-offs, with transfer learning experiments suggesting practical applicability to larger datasets like ImageNet.

Abstract

This paper proposes a neural architecture search space using ResNet as a framework, with search objectives including parameters for convolution, pooling, fully connected layers, and connectivity of the residual network. In addition to recognition accuracy, this paper uses the loss value on the validation set as a secondary objective for optimization. The experimental results demonstrate that the search space of this paper together with the optimisation approach can find competitive network architectures on the MNIST, Fashion-MNIST and CIFAR100 datasets.

Figures (4)

• Figure 1: Illustration of search space. The fixed part and the ResNet block consist of convolutional layers and pooling layers. In the process of initialization, the number of ResNet blocks N and the number of FFN layers M are randomly generated within a preset range. Then, the number of convolutional and pooling layers in each ResNet block is also randomly generated within another preset range. Last but not least, the hyperparameters of each conv/pool/FFN layer are also randomly selected within a predefined range. Since for every individual, the fixed part will be the same, it does not need to be encoded. The encoding strategy of the individual could be recursively defined using pseudo regular expressions as follows {\bf gene\_encode}\rightarrow {\bf resnet\_block\_encode}^N\ {\bf FFN\_layer\_encode}^M{\bf resnet\_block\_encode}\rightarrow ({\bf conv\_layer\_encode} \ \| \ {\bf pool\_layer\_encode})^+{\bf conv\_layer\_encode}\rightarrow [{\bf conv\_identify},\ (filter\_width,\ filter\_height), (stride\_width,\ stride\_height),\ output\_channels]{\bf pool\_layer\_encode}\rightarrow [{\bf pool\_identify},\ (filter\_width,\ filter\_height), (stride\_width,\ stride\_height),\ pool\_type]{\bf FFN\_layer\_encode}\rightarrow [{\bf FFN\_identify},\ output\_neurons]{\bf conv\_identify}\rightarrow 1,\ {\bf pool\_identify} \rightarrow 2,\ {\bf FFN\_identify} \rightarrow 3The encoding position $pool\_type$ indicates maximum or average pooling. Note that in a ResNet block, when the input and the output have different lengths, widths, or number of channels, a $1\times1$ conv is required to downsample the input.
• Figure 2: Illustration of genetic operators between individuals. (a)(b)(c) show the crossover, while (d)(e)(f) show the mutation.
• Figure 3: A brief illustration of BenchENAS platform deployment. The central node server is composed of the controller part and redis. After the individuals of each population are generated, the controller distributes all individual neural networks to the worker nodes. In the training process, the controller monitors the GPU status of the working node. When a GPU with sufficient video memory and suitable for training appears, a work node is ordered to create a process on the GPU to train an individual neural network. After the training is completed, the work node reports the result to the redis of the central node. After receiving the redis write event, the central node records the reported data in a disk file. According to the training results, the central node makes selection, crossover and mutation operations. When the training process of an individual ends, the GPU memory occupied will be released and allocated to the next individual. The central node server does not perform specific training tasks.
• Figure 4: Figure of the change trend of error, loss and ratio (i.e. $error/loss$) values within 15 epochs.