Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

FX-DARTS: Designing Topology-unconstrained Architectures with Differentiable Architecture Search and Entropy-based Super-network Shrinking

Xuan Rao, Bo Zhao, Derong Liu, Cesare Alippi

TL;DR

FX-DARTS addresses the limitations of traditional DARTS by removing topology-sharing constraints and introducing an entropy-based super-network shrinking framework. ESS decouples super-network training from architecture optimization and uses adaptive entropy regularization, warm-up phases, and cyclic reinitialization to derive diverse, discrete architectures within a single search. Empirical results on CIFAR-10/100, TinyImageNet, and ImageNet show competitive accuracy and efficiency across multiple operator spaces and task regimes, highlighting the potential of topology-unconstrained NAS. The work also provides theoretical insights into the convergence behavior of sparsity entropy and analyzes the impact of ablations, while acknowledging the challenges posed by the enlarged search space and outlining directions for future improvements in Auto-ML automation.

Abstract

Strong priors are imposed on the search space of Differentiable Architecture Search (DARTS), such that cells of the same type share the same topological structure and each intermediate node retains two operators from distinct nodes. While these priors reduce optimization difficulties and improve the applicability of searched architectures, they hinder the subsequent development of automated machine learning (Auto-ML) and prevent the optimization algorithm from exploring more powerful neural networks through improved architectural flexibility. This paper aims to reduce these prior constraints by eliminating restrictions on cell topology and modifying the discretization mechanism for super-networks. Specifically, the Flexible DARTS (FX-DARTS) method, which leverages an Entropy-based Super-Network Shrinking (ESS) framework, is presented to address the challenges arising from the elimination of prior constraints. Notably, FX-DARTS enables the derivation of neural architectures without strict prior rules while maintaining the stability in the enlarged search space. Experimental results on image classification benchmarks demonstrate that FX-DARTS is capable of exploring a set of neural architectures with competitive trade-offs between performance and computational complexity within a single search procedure.

FX-DARTS: Designing Topology-unconstrained Architectures with Differentiable Architecture Search and Entropy-based Super-network Shrinking

TL;DR

FX-DARTS addresses the limitations of traditional DARTS by removing topology-sharing constraints and introducing an entropy-based super-network shrinking framework. ESS decouples super-network training from architecture optimization and uses adaptive entropy regularization, warm-up phases, and cyclic reinitialization to derive diverse, discrete architectures within a single search. Empirical results on CIFAR-10/100, TinyImageNet, and ImageNet show competitive accuracy and efficiency across multiple operator spaces and task regimes, highlighting the potential of topology-unconstrained NAS. The work also provides theoretical insights into the convergence behavior of sparsity entropy and analyzes the impact of ablations, while acknowledging the challenges posed by the enlarged search space and outlining directions for future improvements in Auto-ML automation.

Abstract

Strong priors are imposed on the search space of Differentiable Architecture Search (DARTS), such that cells of the same type share the same topological structure and each intermediate node retains two operators from distinct nodes. While these priors reduce optimization difficulties and improve the applicability of searched architectures, they hinder the subsequent development of automated machine learning (Auto-ML) and prevent the optimization algorithm from exploring more powerful neural networks through improved architectural flexibility. This paper aims to reduce these prior constraints by eliminating restrictions on cell topology and modifying the discretization mechanism for super-networks. Specifically, the Flexible DARTS (FX-DARTS) method, which leverages an Entropy-based Super-Network Shrinking (ESS) framework, is presented to address the challenges arising from the elimination of prior constraints. Notably, FX-DARTS enables the derivation of neural architectures without strict prior rules while maintaining the stability in the enlarged search space. Experimental results on image classification benchmarks demonstrate that FX-DARTS is capable of exploring a set of neural architectures with competitive trade-offs between performance and computational complexity within a single search procedure.
Paper Structure (29 sections, 3 theorems, 24 equations, 5 figures, 5 tables, 2 algorithms)

This paper contains 29 sections, 3 theorems, 24 equations, 5 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related Works
  3. Neural architecture search
  4. Flexible architecture search
  5. Preliminary
  6. Cell-based Convolution Neural Architectures
  7. Continuous Relaxation of Search Space
  8. Topology-Constrained Super-network Optimization and Discretization
  9. Flexible Differentiable Architecture Search
  10. Breaking the Topological Constraints in DARTS
  11. Flexible DARTS
  12. Information Entropy Loss
  13. Feedback-based Adaptive Coefficient Adjustment
  14. Super-network Warm-up Phase
  15. Architectural Optimization Phase
  16. ...and 14 more sections

Key Result

Corollary 1

The gradient $\nabla_{\alpha} \mathcal{H}_{(j)}^{\rm node} = \left[ \frac{\partial \mathcal{H}_{(j)}^{\rm node}}{\partial \alpha_{(i,j)}^o} \right]$ satisfies Specifically, when $P_{j}$ is not one-hot, $\nabla_{\alpha} \mathcal{H}_{(j)}^{\rm node}$ is non-zero. This implies that $\nabla_{\alpha} \mathcal{H}_{(j)}^{\rm node} = 0$ is achieved when $\mathcal{H}_{(j)}^{\rm node} = 0$ only. Furthermor

Figures (5)

  • Figure 1: Visualization of topology-constrained architectures and our flexible architectures derived on the CIFAR dataset. (a) Search space comparison. Specifically, topology-constrained architectures are included in the topology-unconstrained search space. (b) and (c) Architectures derived in the topology-constrained search space by DARTS and CDARTS, respectively. These methods assume that cells of the same type share the same structure. (d) and (e) Architectures derived by our FX-DARTS in the topology-unconstrained search space in one NAS procedure. Particularly, all cells have their unique structures.
  • Figure 2: The networks searched by FX-DARTS on CIFAR-10. Green nodes are the input, pre-processed tensors and the output. Light-blue nodes and dark-red nodes are the intermediate nodes for normal cells and reduction cells, respectively. Grey dashed lines indicate the concatenation of feature tensors at channel dimension. In particular, red directed lines are skip-connections, blue directed lines are $3\times3$ depthwise separable convolutions, and green directed lines are $3\times3$ dilated convolutions.
  • Figure 3: Performance comparison of FX-DARTS vs. Topology-constrained NAS architectures on CIFAR-10/100 classification tasks. The results correspond to Table \ref{['tab:results_CIFAR-10']}.
  • Figure 4: Evolution of cell-level sparsity entropy during the architecture search process. As shown, the gradually decreased sparsity entropy of each cell indicates that the super-network structure becomes increasingly sparse.
  • Figure 5: Performance comparison of FX-DARTS vs. Topology-constrained architectures on the multi-task evaluations. The results correspond to Table \ref{['tab:model_performance_updated']}. Specifically, The figure illustrates the relative accuracy improvement (in percentage points) compared to the mean accuracy of all models in each dataset.

Theorems & Definitions (10)

  • Remark 1
  • proof
  • Corollary 1
  • proof
  • Theorem 1
  • proof
  • Remark 2
  • Corollary 2
  • proof
  • Remark 3