Towards Accurate and Robust Architectures via Neural Architecture Search
Yuwei Ou, Yuqi Feng, Yanan Sun
TL;DR
This work tackles the challenge that neural architecture choices constrain robustness under adversarial training. It introduces ARNAS, which combines an Accurate and Robust search space with a differentiable multi-objective optimization (MGDA-based) to jointly minimize natural and adversarial losses during search. Empirical results across CIFAR-10/100, SVHN, and Tiny-ImageNet show ARNAS achieves strong adversarial robustness with competitive natural accuracy and good transferability, outperforming prior robust NAS methods. The findings reveal that deploying different cell types at distinct architectural positions—e.g., Accurate Cells near the input and Robust Cells near the output—can jointly enhance accuracy and robustness, offering practical guidance for robust NAS design.
Abstract
To defend deep neural networks from adversarial attacks, adversarial training has been drawing increasing attention for its effectiveness. However, the accuracy and robustness resulting from the adversarial training are limited by the architecture, because adversarial training improves accuracy and robustness by adjusting the weight connection affiliated to the architecture. In this work, we propose ARNAS to search for accurate and robust architectures for adversarial training. First we design an accurate and robust search space, in which the placement of the cells and the proportional relationship of the filter numbers are carefully determined. With the design, the architectures can obtain both accuracy and robustness by deploying accurate and robust structures to their sensitive positions, respectively. Then we propose a differentiable multi-objective search strategy, performing gradient descent towards directions that are beneficial for both natural loss and adversarial loss, thus the accuracy and robustness can be guaranteed at the same time. We conduct comprehensive experiments in terms of white-box attacks, black-box attacks, and transferability. Experimental results show that the searched architecture has the strongest robustness with the competitive accuracy, and breaks the traditional idea that NAS-based architectures cannot transfer well to complex tasks in robustness scenarios. By analyzing outstanding architectures searched, we also conclude that accurate and robust neural architectures tend to deploy different structures near the input and output, which has great practical significance on both hand-crafting and automatically designing of accurate and robust architectures.