Hierarchical Representations for Efficient Architecture Search

TL;DR

The paper tackles the efficiency of neural architecture search by introducing hierarchical representations that compose architectures from reusable motifs. It shows that simple search strategies, including random search and evolution, can find competitive image-classification cells when paired with a powerful hierarchical encoding. Empirically, hierarchical evolution yields the best CIFAR-10 results among evolutionary methods (3.75% top-1) and competitive ImageNet performance (20.3% top-1) with 64M parameters. The work demonstrates substantial speedups and scalable search without resorting to resource-intensive reinforcement learning or weight-sharing strategies.

Abstract

We explore efficient neural architecture search methods and show that a simple yet powerful evolutionary algorithm can discover new architectures with excellent performance. Our approach combines a novel hierarchical genetic representation scheme that imitates the modularized design pattern commonly adopted by human experts, and an expressive search space that supports complex topologies. Our algorithm efficiently discovers architectures that outperform a large number of manually designed models for image classification, obtaining top-1 error of 3.6% on CIFAR-10 and 20.3% when transferred to ImageNet, which is competitive with the best existing neural architecture search approaches. We also present results using random search, achieving 0.3% less top-1 accuracy on CIFAR-10 and 0.1% less on ImageNet whilst reducing the search time from 36 hours down to 1 hour.

Figures (11)

• Figure 1: An example of a three-level hierarchical architecture representation. The bottom row shows how level-1 primitive operations $o_1^{(1)}, o_2^{(1)}, o_3^{(1)}$ are assembled into a level-2 motif $o_1^{(2)}$. The top row shows how level-2 motifs $o_1^{(2)}, o_2^{(2)}, o_3^{(2)}$ are then assembled into a level-3 motif $o_1^{(3)}$.
• Figure 2: Image classification models constructed using the cells optimized with architecture search. Top-left: small model used during architecture search on CIFAR-10. Top-right: large CIFAR-10 model used for learned cell evaluation. Bottom: ImageNet model used for learned cell evaluation.
• Figure 3: Fitness and number of parameters vs evolution step for flat and hierarchical representations. Left: fitness of a genotype generated at each evolution step. Middle: maximum fitness across all genotypes generated before each evolution step. Right: number of parameters in the small CIFAR-10 model constructed using the genotype generated at each evolution step.
• Figure 4: Accuracy improvement over the course of evolution, measured with respect to the first random genotype. The small model is the model used for fitness computation during evolution (its absolute fitness value is shown with the red curve in Fig. \ref{['fig:fitness_evo']} (middle)). The large model is the model where the evolved cell architecture is deployed for training and evaluation.
• Figure 5: Cell