Robust Principles: Architectural Design Principles for Adversarially Robust CNNs

TL;DR

This work tackles the inconsistency in how CNN architectural choices affect adversarial robustness by proposing three generalizable principles: (A) optimal depth–width ranges, (B) convolutional stem over patch-based downsampling, and (C) robust residual blocks with squeeze-and-excitation and non-parametric smooth activations. Through extensive experiments on CIFAR-10, CIFAR-100, and ImageNet across multiple training recipes and model families, the authors show consistent robustness gains, culminating in ra architectures that outperform strong baselines including Transformers and NAS-based networks under adversarial evaluation. The findings demonstrate that principled architectural design can substantially enhance robustness, not just training tricks, and provide practical guidelines for building more resilient CNNs. The work is supported by public code and broad empirical validation across dataset scales and design spaces.

Abstract

Our research aims to unify existing works' diverging opinions on how architectural components affect the adversarial robustness of CNNs. To accomplish our goal, we synthesize a suite of three generalizable robust architectural design principles: (a) optimal range for depth and width configurations, (b) preferring convolutional over patchify stem stage, and (c) robust residual block design through adopting squeeze and excitation blocks and non-parametric smooth activation functions. Through extensive experiments across a wide spectrum of dataset scales, adversarial training methods, model parameters, and network design spaces, our principles consistently and markedly improve AutoAttack accuracy: 1-3 percentage points (pp) on CIFAR-10 and CIFAR-100, and 4-9 pp on ImageNet. The code is publicly available at https://github.com/poloclub/robust-principles.

Figures (3)

• Figure 1: We synthesize a suite of generalizable architectural design principles to robustify cnn, spanning a network's macro and micro designs: (A) optimal range for depth and width configurations, (B) preferring convolutional over patchify stem stage, and (C) robust residual block design by adopting se blocks, and non-parametric smooth activation functions. The principles consistently and markedly improve aa accuracy for CIFAR-10, CIFAR-100, and ImageNet over the wide spectrum of at methods, model parameters, and network design spaces.
• Figure 2: (a) Clean and pgd accuracies are negatively correlated with the wd ratio. Each dot is a configuration of #stage, depth, and width. Intersecting each attack budget's top 10% most accurate configurations, we find the optimal range of the wd ratio is $[7.5, 13.5]$ (with color backgrounds) and verify the significance by the edf of models within range (steeper dark lines) and outside of range (gentle light lines). (b) Performance of different configurations in stem stage and residual blocks. Differences are compared with baseline ResNet-50. All models trained with Fast-at wong2020fast and evaluated on full ImageNet validation set. Different pgd attack budgets show a similar accuracy trend, and the full results are shown in supplementary material Sec. \ref{['sec:appx-other-components']}.
• Figure 3: Visualization of the Fast-at curves of individual architectural modifications (Table \ref{['tab:roadmap']}), and the sat curves of the final robustified model (Table \ref{['tab:results-imagenet']}). We observed that a lower training loss leads to a higher robustness as expected and no catastrophic overfitting occurs during training.