POMONAG: Pareto-Optimal Many-Objective Neural Architecture Generator

TL;DR

POMONAG addresses the need for Pareto-efficient, transferable neural architecture search by extending DiffusionNAG to a many-objective diffusion process that optimizes accuracy, parameter count, MACs, and latency. It introduces many-objective guidance via Performance Predictors, expands the Meta-Dataset, and applies Pareto Front Filtering and Stretching to produce a diverse set of Pareto-optimal architectures. The approach is validated on NASBench201 and MobileNetV3 across 15 image classification datasets, achieving higher accuracy and improved efficiency with fewer trained models than prior state-of-the-art. This work enables more scalable deployment of NAS-generated models in resource-constrained environments and suggests future extensions to segmentation and detection tasks to broaden applicability.

Abstract

Neural Architecture Search (NAS) automates neural network design, reducing dependence on human expertise. While NAS methods are computationally intensive and dataset-specific, auxiliary predictors reduce the models needing training, decreasing search time. This strategy is used to generate architectures satisfying multiple computational constraints. Recently, Transferable NAS has emerged, generalizing the search process from dataset-dependent to task-dependent. In this field, DiffusionNAG is a state-of-the-art method. This diffusion-based approach streamlines computation, generating architectures optimized for accuracy on unseen datasets without further adaptation. However, by focusing solely on accuracy, DiffusionNAG overlooks other crucial objectives like model complexity, computational efficiency, and inference latency -- factors essential for deploying models in resource-constrained environments. This paper introduces the Pareto-Optimal Many-Objective Neural Architecture Generator (POMONAG), extending DiffusionNAG via a many-objective diffusion process. POMONAG simultaneously considers accuracy, number of parameters, multiply-accumulate operations (MACs), and inference latency. It integrates Performance Predictor models to estimate these metrics and guide diffusion gradients. POMONAG's optimization is enhanced by expanding its training Meta-Dataset, applying Pareto Front Filtering, and refining embeddings for conditional generation. These enhancements enable POMONAG to generate Pareto-optimal architectures that outperform the previous state-of-the-art in performance and efficiency. Results were validated on two search spaces -- NASBench201 and MobileNetV3 -- and evaluated across 15 image classification datasets.

Figures (4)

• Figure 1: Overview of the POMONAG framework illustrating the Many-Objective Reverse Diffusion Guidance process.
• Figure 2: Distributions of key metrics in the proposed Meta-Datasets. Histograms display accuracy, number of parameters, MACs, and latency for architectures sampled from the NASBench201 (top row, 10,000 architectures) and MobileNetV3 (bottom row, 20,000 architectures) search spaces. The architectures were randomly selected and trained on subsets of ImageNet32, demonstrating the diversity within each search space.
• Figure 3: Performance comparison of architectures generated by DiffusionNAG and POMONAG on 15 datasets using the NASBench201 search space. Each plot illustrates accuracy versus a secondary metric (number of parameters, MACs, or latency). The blue lines represent POMONAG's Pareto Fronts, with circles highlighting the efficient (yellow), balanced (purple), and accurate (green) variants. Orange points depict DiffusionNAG results. Error bars show 95% confidence intervals averaged over three runs. Dashed lines indicate DiffusionNAG results as reported by An et al. an2024diffusionnag.
• Figure 4: Performance comparison of architectures generated by DiffusionNAG and POMONAG on 15 datasets using the MobileNetV3 search space. Each plot illustrates accuracy versus a secondary metric (number of parameters, MACs, or latency). The blue lines represent POMONAG's Pareto Fronts, with circles highlighting the efficient (yellow), balanced (purple), and accurate (green) variants. Orange points depict DiffusionNAG results. Error bars show 95% confidence intervals averaged over three runs. Dashed lines indicate DiffusionNAG results as reported by An et al. an2024diffusionnag.