Multi-Objective Neural Architecture Search by Learning Search Space Partitions

TL;DR

This work introduces LaMOO, a learning-based meta-optimizer for multi-objective neural architecture search (NAS) that partitions the search space into promising and non-promising regions and focuses sampling within these regions using Monte Carlo Tree Search. By combining space-partition learning with sampling strategies (random, qEHVI, CMA-ES) and integrating with one-shot, few-shot, and predictor-based NAS, LaMOO achieves substantial gains in sample efficiency and Pareto-front quality across NasBench201, NasBench301, HW-NAS-Bench, and open-domain tasks (CIFAR-10, ImageNet, Penn Treebank, MS COCO). The approach leverages dominance-based region labeling, SVM-based partitions, and hypervolume as a guiding metric, while exploring leaf-based MCTS to reduce hypervolume computation costs in many-objective settings. Empirically, LaMOO yields state-of-the-art or competitive performance with dramatically fewer samples and lower resource requirements, demonstrating strong practical impact for hardware-aware and open-domain NAS. The work also analyzes design choices (sampling methods, kernel types, and MCTS strategies) and suggests avenues for extending space-partitioning techniques to other AutoML and ML-design problems.

Abstract

Deploying deep learning models requires taking into consideration neural network metrics such as model size, inference latency, and #FLOPs, aside from inference accuracy. This results in deep learning model designers leveraging multi-objective optimization to design effective deep neural networks in multiple criteria. However, applying multi-objective optimizations to neural architecture search (NAS) is nontrivial because NAS tasks usually have a huge search space, along with a non-negligible searching cost. This requires effective multi-objective search algorithms to alleviate the GPU costs. In this work, we implement a novel multi-objectives optimizer based on a recently proposed meta-algorithm called LaMOO on NAS tasks. In a nutshell, LaMOO speedups the search process by learning a model from observed samples to partition the search space and then focusing on promising regions likely to contain a subset of the Pareto frontier. Using LaMOO, we observe an improvement of more than 200% sample efficiency compared to Bayesian optimization and evolutionary-based multi-objective optimizers on different NAS datasets. For example, when combined with LaMOO, qEHVI achieves a 225% improvement in sample efficiency compared to using qEHVI alone in NasBench201. For real-world tasks, LaMOO achieves 97.36% accuracy with only 1.62M #Params on CIFAR10 in only 600 search samples. On ImageNet, our large model reaches 80.4% top-1 accuracy with only 522M #FLOPs.

Figures (16)

• Figure 1: The visualization of search space for the Branin-Currin. Red samples have low dominance numbers, while blue ones have high dominance numbers. The smaller the dominance number, the better the quality of the sample.
• Figure 2: 500 samples on Nasbench201 nasbench201 searched by the covariance matrix adaptation evolution strategy (CMA-ES) cmacma-es with single and multi-objective versions. (a), (b) and (c) plot accuracy vs. #FLOPs of samples in the Pareto frontier. (d) is the contour lines with a sclarized objective.
• Figure 3: The overview of a LaMOO iteration. The left portion depicts the Learning Space Partition phase for optimizing two objectives. The first figure depicts the value space $(f_1, f_2)$ and visualizes the hypervolume $HV$ (blue-shaded area) given the Pareto frontier $P$ and the reference point $R$. The middle figure shows the search space $\Omega$ and its partitions (i.e., $\Omega_{C}$, $\Omega_{D}$, and $\Omega_{E}$) based on samples collected from the previous iterations and their dominance numbers in the objective space. The right figure shows the tree constructed based on the partitions. The right portion depicts the Sampling in Selected Regions phase. The left figure visualizes two selection strategies described in §\ref{['sec:monte_carlo_tree_search']}. The right figure shows that new architectures will be sampled from the good partition $\Omega_{E}$ with any sampling algorithms. Figure adapted from our prior work lamoo.
• Figure 4: Visualization of example hypervolume calculation.
• Figure 5: The range of hypervolume for 50 samples randomly generated from different regions in NasBench201, NasBench201 by supernet search, and NasBench301.