Is ReLU Adversarially Robust?

TL;DR

Is ReLU Adversarially Robust? investigates whether standard ReLU activations contribute to adversarial vulnerability and proposes a capped-ReLU to curb perturbation growth across layers. The method formalizes the cap as $a(z, \beta) = \max(0, \min(z, \beta))$ and demonstrates, on MNIST with PGD attacks, that capping improves robustness; the robustness varies with cap placement and network architecture. Adversarial training with capped-ReLU further strengthens robustness beyond conventional adversarial training, though vanishing-gradient issues limit applicability to very large models. The work offers a new activation-function design direction for practical robustness and suggests future work to scale and address remaining attacks like CW.

Abstract

The efficacy of deep learning models has been called into question by the presence of adversarial examples. Addressing the vulnerability of deep learning models to adversarial examples is crucial for ensuring their continued development and deployment. In this work, we focus on the role of rectified linear unit (ReLU) activation functions in the generation of adversarial examples. ReLU functions are commonly used in deep learning models because they facilitate the training process. However, our empirical analysis demonstrates that ReLU functions are not robust against adversarial examples. We propose a modified version of the ReLU function, which improves robustness against adversarial examples. Our results are supported by an experiment, which confirms the effectiveness of our proposed modification. Additionally, we demonstrate that applying adversarial training to our customized model further enhances its robustness compared to a general model.

Figures (11)

• Figure 1: The distance between each hidden layer's outputs resulted from passing clean samples and adversarial examples. Note that the top row shows the $L_\infty$ distance and the bottom row shows the $L_2$ distance.
• Figure 2: Accuracy achieved by classifiers with different capped hidden layers and max values on MNIST test dataset.
• Figure 3: Sigmoid functions with different constant $c$.
• Figure 4: Standard accuracy, robust accuracy and success rate of the two-hidden-layer classifier according to PGD attack over a range of max values.
• Figure 5: Standard accuracy, robust accuracy and success rate of the reversed two-hidden-layer classifier according to PGD attack over a range of max values.