Efficient Multi-Objective Neural Architecture Search via Pareto Dominance-based Novelty Search

TL;DR

The paper tackles NAS by addressing biases and inefficiencies of objective-only search, introducing MTF-PDNS, a Pareto Dominance-based Novelty Search that uses multiple training-free metrics (synflow, jacov, snip) plus a complexity descriptor to drive exploration. By maintaining an elitist archive of non-dominated architectures and calculating a dynamic novelty score relative to this archive, the approach balances exploration and quality without training-weighted objectives. Empirical results across NAS benchmarks show faster convergence, greater architectural diversity, and strong transferability, often achieving comparable or superior front quality at dramatically reduced computational cost compared to training-based methods. This work demonstrates that a multi-metric novelty-driven search can effectively navigate the NAS space, offering practical benefits for rapid and robust architecture discovery.

Abstract

Neural Architecture Search (NAS) aims to automate the discovery of high-performing deep neural network architectures. Traditional objective-based NAS approaches typically optimize a certain performance metric (e.g., prediction accuracy), overlooking large parts of the architecture search space that potentially contain interesting network configurations. Furthermore, objective-driven population-based metaheuristics in complex search spaces often quickly exhaust population diversity and succumb to premature convergence to local optima. This issue becomes more complicated in NAS when performance objectives do not fully align with the actual performance of the candidate architectures, as is often the case with training-free metrics. While training-free metrics have gained popularity for their rapid performance estimation of candidate architectures without incurring computation-heavy network training, their effective incorporation into NAS remains a challenge. This paper presents the Pareto Dominance-based Novelty Search for multi-objective NAS with Multiple Training-Free metrics (MTF-PDNS). Unlike conventional NAS methods that optimize explicit objectives, MTF-PDNS promotes population diversity by utilizing a novelty score calculated based on multiple training-free performance and complexity metrics, thereby yielding a broader exploration of the search space. Experimental results on standard NAS benchmark suites demonstrate that MTF-PDNS outperforms conventional methods driven by explicit objectives in terms of convergence speed, diversity maintenance, architecture transferability, and computational costs.

Figures (4)

• Figure 1: Illustration of Pareto Dominance-based Novelty Search. In the elitist archive update (yellow background), if we use descriptors derived from training-based metrics to calculate novelty scores, the x-axis represents validation error. In contrast, if we use descriptors derived from training-free metrics, the x-axis represents training-free metrics as shown in the figure.
• Figure 2: Distribution of operations in architectures on final approximation fronts of MTF-MOENAS and MTF-PDNS on NAS-Bench-201.
• Figure 3: Comparison of IGD$^+$ (top) and Hypervolume (bottom) metrics with respect to GPU hours on NAS benchmarks.
• Figure 4: Visual representation of architectures explored by MTF-MOENAS (top) and MTF-PDNS (bottom) within the performance and complexity space on NAS benchmarks. Circles with no edges and blurred represent the architectures that are not explored by the corresponding search algorithm.