Guardian of the Ensembles: Introducing Pairwise Adversarially Robust Loss for Resisting Adversarial Attacks in DNN Ensembles

TL;DR

This paper addresses adversarial robustness in deep neural networks by focusing on ensemble diversity. It introduces PARL, a loss that enforces pairwise diversity across ensemble members through gradient-based and intermediate representation metrics, producing dissimilar decision boundaries. Empirical results on CIFAR-10/100 and Tiny Imagenet show PARL substantially improves robust accuracy against black-box transfer and query-based attacks while maintaining clean accuracy and reducing training time relative to prior ensemble defenses and adversarial training. The findings demonstrate that fostering diversity across both decision boundaries and internal representations can meaningfully resist adversarial perturbations in practical settings, with strong performance across multiple architectures and datasets.

Abstract

Adversarial attacks rely on transferability, where an adversarial example (AE) crafted on a surrogate classifier tends to mislead a target classifier. Recent ensemble methods demonstrate that AEs are less likely to mislead multiple classifiers in an ensemble. This paper proposes a new ensemble training using a Pairwise Adversarially Robust Loss (PARL) that by construction produces an ensemble of classifiers with diverse decision boundaries. PARL utilizes outputs and gradients of each layer with respect to network parameters in every classifier within the ensemble simultaneously. PARL is demonstrated to achieve higher robustness against black-box transfer attacks than previous ensemble methods as well as adversarial training without adversely affecting clean example accuracy. Extensive experiments using standard Resnet20, WideResnet28-10 classifiers demonstrate the robustness of PARL against state-of-the-art adversarial attacks. While maintaining similar clean accuracy and lesser training time, the proposed architecture has a 24.8% increase in robust accuracy ($ε$ = 0.07) from the state-of-the art method.

Figures (12)

• Figure 1: (a) Input image; (b) $\nabla_{prim}$: Gradient of loss in the primary model; (c) $\nabla_{sim}$: Gradient of loss in another model with similar decision boundaries; (d) $\nabla_{div}$: Gradient of loss in a model with not so similar decision boundaries but comparable accuracy; (e) Symbolic directions of all the gradients in higher dimensions. Gradients are computed with respect to the image shown in (a).
• Figure 2: (a) Input image; (b) and (c) Similar outputs of an intermediate convolution layer of two classifiers with similar internal representations; (d) and (e) Contrasting outputs of an intermediate convolution layer of two classifiers which are trained simultaneously to have distinct internal representations
• Figure 3: Layer-wise linear CKA values between each pair of models trained with CIFAR-10 on (a) Resnet20 and (b) WideResnet28-10 showing the similarities at each layer.
• Figure 4: Resnet20 Ensemble classification accuracy (%) vs. Attack Strength ($\epsilon$) against black-box transfer attacks generated from surrogate ensemble with (a) CIFAR-10 and (b) CIFAR-100 dataset
• Figure 5: (a) Resnet20 (RN20) and WideResnet28-10 (WRN28-10) Ensemble classification accuracy (%) vs. Attack Strength ($\epsilon$) against Square attack for CIFAR-10 (b) Average number of queries required for square attack for all samples as well as successful (succ) attack samples with RN20