Multiple Population Alternate Evolution Neural Architecture Search

TL;DR

This work tackles the inefficiency and limited diversity in neural architecture search by introducing Multiple Population Alternate Evolution (MPAE). It partitions the search space into $L$ interconnected units and evolves multiple populations alternately, augmented with a migration mechanism that transfers knowledge between populations via a migration archive. The framework relies on a shared supernet with weight sharing and subnetwork sampling to evaluate architectures efficiently, achieving state-of-the-art results on CIFAR with only $0.3$ GPU-days and competitive ImageNet performance. Overall, MPAE demonstrates that dividing the search into smaller subspaces and leveraging cross-population knowledge can dramatically reduce search costs while maintaining architectural diversity and performance gains.

Abstract

The effectiveness of Evolutionary Neural Architecture Search (ENAS) is influenced by the design of the search space. Nevertheless, common methods including the global search space, scalable search space and hierarchical search space have certain limitations. Specifically, the global search space requires a significant amount of computational resources and time, the scalable search space sacrifices the diversity of network structures and the hierarchical search space increases the search cost in exchange for network diversity. To address above limitation, we propose a novel paradigm of searching neural network architectures and design the Multiple Population Alternate Evolution Neural Architecture Search (MPAE), which can achieve module diversity with a smaller search cost. MPAE converts the search space into L interconnected units and sequentially searches the units, then the above search of the entire network be cycled several times to reduce the impact of previous units on subsequent units. To accelerate the population evolution process, we also propose the the population migration mechanism establishes an excellent migration archive and transfers the excellent knowledge and experience in the migration archive to new populations. The proposed method requires only 0.3 GPU days to search a neural network on the CIFAR dataset and achieves the state-of-the-art results.

Figures (4)

• Figure 1: Overall process of MPAE. Step 1 of the algorithm is to initialize multiple populations and immigrant archives. Step 2 is to generate offspring populations and merge them with parent populations and migrating populations. Step 3 involves select the parent populations for the next generation through multi-objective environmental selection and updating the immigrant archives. Step 4 is when the populations inherit the weights of the super network during evolution and update the weights of the super network after training for a few epochs. Finally, the optimal network is obtained through the search process.
• Figure 2: Multi-Populations Initialization. Individual encode refers to the transformation of networks sampled from the search space into a two-dimensional matrix through encode strategy. The initialization of population $P_i$ is the process of generating $N_i$ network individuals. The initialization of multiple populations is the process of creating $L$ populations based on the number of layers $L$ in the complete network. It is important to note that the $L$ individuals sampled from multiple populations together constitute a complete network.
• Figure 3: Search efficiency comparison between MPAE and other baselines in terms of (a) validation accuracy and (b) required compute time in GPU-Days. The search cost is measured on CIFAR-10 for most methods, except Block-QNN, where the CIFAR-100 dataset is used for.
• Figure 4: Comparison graph of migrated individuals and offspring individuals at various stages of the MPAE evolutionary process.