TopoNAS: Boosting Search Efficiency of Gradient-based NAS via Topological Simplification

TL;DR

TopoNAS addresses the efficiency bottlenecks of gradient-based one-shot NAS by explicitly modeling and simplifying the topology of searchable paths, coupled with a kernel normalization strategy to stabilize reparameterization in nonlinear search spaces. It introduces Partial Module Sharing and Floating Module Sharing to recursively reduce redundant computation and enable effective reparameterization, yielding significant memory and time savings across NASBench201 and DARTS-based baselines. Empirical results show notable accuracy improvements (e.g., up to around 9 percentage points on IN-16 in some configurations) and substantial reductions in normalized search cost, while maintaining or improving retraining performance. By acting as a model-agnostic plug-in, TopoNAS broadens the applicability of gradient-based one-shot NAS to nonlinear search spaces with improved stability and efficiency, though it introduces some overhead in simple spaces and may create gaps between searching and retraining that warrant further refinement.

Abstract

Improving search efficiency serves as one of the crucial objectives of Neural Architecture Search (NAS). However, many current approaches ignore the universality of the search strategy and fail to reduce the computational redundancy during the search process, especially in one-shot NAS architectures. Besides, current NAS methods show invalid reparameterization in non-linear search space, leading to poor efficiency in common search spaces like DARTS. In this paper, we propose TopoNAS, a model-agnostic approach for gradient-based one-shot NAS that significantly reduces searching time and memory usage by topological simplification of searchable paths. Firstly, we model the non-linearity in search spaces to reveal the parameterization difficulties. To improve the search efficiency, we present a topological simplification method and iteratively apply module-sharing strategies to simplify the topological structure of searchable paths. In addition, a kernel normalization technique is also proposed to preserve the search accuracy. Experimental results on the NASBench201 benchmark with various search spaces demonstrate the effectiveness of our method. It proves the proposed TopoNAS enhances the performance of various architectures in terms of search efficiency while maintaining a high level of accuracy. The project page is available at https://xdedss.github.io/topo_simplification.

Figures (8)

• Figure 1: TopoNAS improves search efficiency in terms of time cost and memory cost with various gradient-based NAS baselines. $\beta$-DARTS denotes searching with standard NASBench201 settings. $\beta$-DARTS* denotes searching with the settings aligned with the original work ye2022beta. $\dagger$ denotes combining $\beta$ regularization proposed by $\beta$-DARTS with PC-DARTS.
• Figure 2: In this graph, each rectangle represents a module to be computed during the search phase while black arrows denote the computing order. TopoNAS enables efficient reparameterization for non-linear search spaces, where existing methods become infeasible due to unaligned levels of non-linearity.
• Figure 3: (a) Parallel topology of an edge. (b) Hierarchical topology after merging shared modules.
• Figure 4: (a) An edge with shared modules between unique modules. (b) The topology after merging shared modules.
• Figure 5: The topology of a simplified edge in DARTS search space.