Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

G-ICSO-NAS: Shifting Gears between Gradient and Swarm for Robust Neural Architecture Search

Xingbang Du, Enzhi Zhang, Rui Zhong, Yang Cao, Masaharu Munetomo

Abstract

Neural Architecture Search (NAS) has become a pivotal technique in automated machine learning. Evolutionary Algorithm (EA)-based methods demonstrate superior search quality but suffer from prohibitive computational costs, while gradient-based approaches like DARTS offer high efficiency but are prone to premature convergence and performance collapse. To bridge this gap, we propose G-ICSO-NAS, a hybrid framework implementing a three-stage optimization strategy. The Warm-up Phase pre-trains supernet weights ($w$) via differentiable methods while architecture parameters ($α$) remain frozen. The Exploration Phase adopts a hybrid co-optimization mechanism: an Improved Competitive Swarm Optimizer (ICSO) with diversity-aware fitness navigates the architecture space to update $α$, while gradient descent concurrently updates $w$. The Stability Phase employs fine-grained gradient-based search with early stopping to converge to the optimal architecture. By synergizing ICSO's global navigation capability with differentiable methods' efficiency, G-ICSO-NAS achieves remarkable performance with minimal cost. In the context of the DARTS search space, an accuracy of 97.46\% is achieved on CIFAR-10 with a computational budget of just 0.15 GPU-Days. The method also exhibits strong transfer potential, recording accuracies of 83.1\% (CIFAR-100) and 75.02\% (ImageNet). Furthermore, regarding the NAS-Bench-201 benchmark, G-ICSO-NAS is shown to deliver state-of-the-art results across all evaluated datasets.

G-ICSO-NAS: Shifting Gears between Gradient and Swarm for Robust Neural Architecture Search

Abstract

Neural Architecture Search (NAS) has become a pivotal technique in automated machine learning. Evolutionary Algorithm (EA)-based methods demonstrate superior search quality but suffer from prohibitive computational costs, while gradient-based approaches like DARTS offer high efficiency but are prone to premature convergence and performance collapse. To bridge this gap, we propose G-ICSO-NAS, a hybrid framework implementing a three-stage optimization strategy. The Warm-up Phase pre-trains supernet weights () via differentiable methods while architecture parameters () remain frozen. The Exploration Phase adopts a hybrid co-optimization mechanism: an Improved Competitive Swarm Optimizer (ICSO) with diversity-aware fitness navigates the architecture space to update , while gradient descent concurrently updates . The Stability Phase employs fine-grained gradient-based search with early stopping to converge to the optimal architecture. By synergizing ICSO's global navigation capability with differentiable methods' efficiency, G-ICSO-NAS achieves remarkable performance with minimal cost. In the context of the DARTS search space, an accuracy of 97.46\% is achieved on CIFAR-10 with a computational budget of just 0.15 GPU-Days. The method also exhibits strong transfer potential, recording accuracies of 83.1\% (CIFAR-100) and 75.02\% (ImageNet). Furthermore, regarding the NAS-Bench-201 benchmark, G-ICSO-NAS is shown to deliver state-of-the-art results across all evaluated datasets.

Paper Structure

This paper contains 17 sections, 20 equations, 2 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Evolution Algorithm for NAS
  4. Differentiable architecture search
  5. Competitive Swarm Optimizer
  6. Methodology
  7. Overview Of G-ICSO-NAS
  8. Warm-Up Stage
  9. Exploration Stage
  10. Stable Stage
  11. Experiment and Result Analysis
  12. Experiment environment
  13. Datasets and Parameters
  14. Experiment Result and Comparison on DARTS search space
  15. Experiment Result and Comparison on NAS-Bench-201 search space
  16. ...and 2 more sections

Figures (2)

  • Figure 1: The overview of G-ICSO-NAS. The warm-up stage trains super-net weights $W$ with unchanged architecture weights $\alpha$; the exploration stage alternates between ICSO-based $\alpha$ updates and gradient-based $W$ updates; the stability stage performs gradient-based optimization with Hoeffding-based early stopping for convergence detection.
  • Figure 2: ICSO. a triplet-based competitive swarm optimizer with diversity-aware fitness evaluation.