Adaptive Channel Allocation for Robust Differentiable Architecture Search

TL;DR

This work addresses the instability of differentiable architecture search (DARTS) caused by the overaccumulation of skip connections during long training, proposing Adaptive Channel Allocation (ACA) to implicitly search and refill skip connections without human priors or extra hyper-parameters. By removing skip connections from the search space and allocating channels in the evaluation stage according to inherited operation strengths, ACA uses BN scaling factors to rank channel importance and assigns more channels to stronger operations, with remaining channels filled by SkipConnect to maintain dimensional consistency. The approach is validated across CIFAR-10 and ImageNet, including standard and complex search spaces, showing improved robustness and accuracy over DARTS and multiple variants, especially under long epochs, and is supported by ablations and visualizations that reveal operation-wise attention and diverse, high-performing architectures. The findings imply that explicit skip-search in the shallow search stage is unnecessary and potentially harmful, and that attention to per-operation channel allocation can robustify differentiable NAS in practical, large-scale settings.

Abstract

Differentiable ARchiTecture Search (DARTS) has attracted much attention due to its simplicity and significant improvement in efficiency. However, the excessive accumulation of the skip connection, when training epochs become large, makes it suffer from weak stability and low robustness, thus limiting its practical applications. Many works have attempted to restrict the accumulation of skip connections by indicators or manual design. These methods, however, are susceptible to human priors and hyper-parameters. In this work, we suggest a more subtle and direct approach that no longer explicitly searches for skip connections in the search stage, based on the paradox that skip connections were proposed to guarantee the performance of very deep networks, but the networks in the search stage of differentiable architecture search are actually very shallow. Instead, by introducing channel importance ranking and channel allocation strategy, the skip connections are implicitly searched and automatically refilled unimportant channels in the evaluation stage. Our method, dubbed Adaptive Channel Allocation (ACA) strategy, is a general-purpose approach for differentiable architecture search, which universally works in DARTS variants without introducing human priors, indicators, or hyper-parameters. Extensive experiments on various datasets and DARTS variants verify that the ACA strategy is the most effective one among existing methods in improving robustness and dealing with the collapse issue when training epochs become large.

Figures (10)

• Figure 1: The overall pipeline of DARTS. (a) Super-net in the search stage. (b) Target-net in the evaluation stage.
• Figure 2: The comparison of DARTS and DARTS-S (DARTS without skip connection in the search stage) on the (a) test error and (b) percentage of skip connections.
• Figure 3: The proposed adaptive channel allocation method. In the target-net, we first employ scaling factors in batch normalization to assess the channel importance. Important channels are allocated to the original operations, and the remaining channels are assigned to skip connections. The number of allocated channels is determined by the operation strength inherited from the search stage.
• Figure 4: Schematic illustration of the cell in the target-net of (a) traditional DARTS variants and (b) the proposed ACA-based variants.
• Figure 5: Test accuracy on CIFAR-10 after searching for large epochs (best view in color).