Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Half Search Space is All You Need

Pavel Rumiantsev, Mark Coates

TL;DR

The paper addresses the memory bottleneck of One-Shot NAS methods like DARTS by proposing a two-stage approach: first prune the NAS search space with Zero-Shot NAS, then apply One-Shot NAS to the reduced space. A Masked One-Shot formulation reduces memory by masking low-importance operations in the mixing layer, while a Zero-Shot pruning step (with a stopping threshold xi) removes clearly poor architectures using a ranking function based on ETENAS. Experiments on the DARTS search space with CIFAR-10/100 show substantial memory and time reductions—up to 81% memory savings—without sacrificing accuracy, and in some cases improving stability. The work demonstrates that large NAS search spaces contain redundancy that can be safely pruned, and that combining Zero-Shot and One-Shot NAS leverages the strengths of both approaches to enable efficient, scalable architecture search.

Abstract

Neural Architecture Search (NAS) is a powerful tool for automating architecture design. One-Shot NAS techniques, such as DARTS, have gained substantial popularity due to their combination of search efficiency with simplicity of implementation. By design, One-Shot methods have high GPU memory requirements during the search. To mitigate this issue, we propose to prune the search space in an efficient automatic manner to reduce memory consumption and search time while preserving the search accuracy. Specifically, we utilise Zero-Shot NAS to efficiently remove low-performing architectures from the search space before applying One-Shot NAS to the pruned search space. Experimental results on the DARTS search space show that our approach reduces memory consumption by 81% compared to the baseline One-Shot setup while achieving the same level of accuracy.

Half Search Space is All You Need

TL;DR

The paper addresses the memory bottleneck of One-Shot NAS methods like DARTS by proposing a two-stage approach: first prune the NAS search space with Zero-Shot NAS, then apply One-Shot NAS to the reduced space. A Masked One-Shot formulation reduces memory by masking low-importance operations in the mixing layer, while a Zero-Shot pruning step (with a stopping threshold xi) removes clearly poor architectures using a ranking function based on ETENAS. Experiments on the DARTS search space with CIFAR-10/100 show substantial memory and time reductions—up to 81% memory savings—without sacrificing accuracy, and in some cases improving stability. The work demonstrates that large NAS search spaces contain redundancy that can be safely pruned, and that combining Zero-Shot and One-Shot NAS leverages the strengths of both approaches to enable efficient, scalable architecture search.

Abstract

Neural Architecture Search (NAS) is a powerful tool for automating architecture design. One-Shot NAS techniques, such as DARTS, have gained substantial popularity due to their combination of search efficiency with simplicity of implementation. By design, One-Shot methods have high GPU memory requirements during the search. To mitigate this issue, we propose to prune the search space in an efficient automatic manner to reduce memory consumption and search time while preserving the search accuracy. Specifically, we utilise Zero-Shot NAS to efficiently remove low-performing architectures from the search space before applying One-Shot NAS to the pruned search space. Experimental results on the DARTS search space show that our approach reduces memory consumption by 81% compared to the baseline One-Shot setup while achieving the same level of accuracy.
Paper Structure (13 sections, 1 equation, 1 figure, 1 table, 1 algorithm)

This paper contains 13 sections, 1 equation, 1 figure, 1 table, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related work
  3. One-Shot architecture search
  4. Zero-Shot architecture search
  5. Problem statement
  6. Methodology
  7. Masked One-Shot
  8. Zero-Shot pruning
  9. Experiments
  10. Dataset and experimental setup
  11. Baselines and techniques
  12. Results
  13. Conclusion

Figures (1)

  • Figure 1: We sampled 100000 architectures from DARTS (has more than $10^{18}$ architectures) and obtained an approximation of their accuracies on CIFAR-10 using surrogate models of NAS-Bench-301 zela2022surrogate. The distribution of architectures (on the left) is heavily skewed towards top-performing architectures. We randomly removed half of the architectures (on the right) to demonstrate the minimal effect on the distribution.