On the Noise Stability and Robustness of Adversarially Trained Networks on NVM Crossbars

TL;DR

This work investigates the robustness of adversarially trained DNNs deployed on NVM crossbar-based analog hardware. By combining adversarial training with the intrinsic non-idealities of crossbars, the authors analyze noise stability (via metrics like Layer Cushion, Interlayer Cushion, Activation Contraction, and Interlayer Smoothness) and show that adversarial training reduces noise stability, leading to greater natural accuracy loss on analog hardware. Under adversarial attacks, however, crossbar non-idealities can confer a measurable robustness gain, particularly for stronger attacks where $\\epsilon_{attack} \\geq \\epsilon_{train}$, with the gain depending on crossbar size and non-ideality factor $NF$. The results emphasize a co-design approach where hardware non-idealities and adversarial training are tuned together to maximize robustness while balancing natural accuracy and energy efficiency; Gaussian-noise baselines alone do not fully capture the hardware-induced robustness observed. Overall, the paper provides practical guidelines for deploying robust DNNs on NVM crossbars and demonstrates the potential of hardware-algorithm co-design to enhance security in energy-efficient ML accelerators.

Abstract

Applications based on Deep Neural Networks (DNNs) have grown exponentially in the past decade. To match their increasing computational needs, several Non-Volatile Memory (NVM) crossbar based accelerators have been proposed. Recently, researchers have shown that apart from improved energy efficiency and performance, such approximate hardware also possess intrinsic robustness for defense against adversarial attacks. Prior works quantified this intrinsic robustness for vanilla DNNs trained on unperturbed inputs. However, adversarial training of DNNs is the benchmark technique for robustness, and sole reliance on intrinsic robustness of the hardware may not be sufficient. In this work, we explore the design of robust DNNs through the amalgamation of adversarial training and intrinsic robustness of NVM crossbar-based analog hardware. First, we study the noise stability of such networks on unperturbed inputs and observe that internal activations of adversarially trained networks have lower Signal-to-Noise Ratio (SNR), and are sensitive to noise compared to vanilla networks. As a result, they suffer on average 2x performance degradation due to the approximate computations on analog hardware. Noise stability analyses show the instability of adversarially trained DNNs. On the other hand, for adversarial images generated using Square Black Box attacks, ResNet-10/20 adversarially trained on CIFAR-10/100 display a robustness gain of 20-30%. For adversarial images generated using Projected-Gradient-Descent (PGD) White-Box attacks, adversarially trained DNNs present a 5-10% gain in robust accuracy due to underlying NVM crossbar when $ε_{attack}$ is greater than $ε_{train}$. Our results indicate that implementing adversarially trained networks on analog hardware requires careful calibration between hardware non-idealities and $ε_{train}$ for optimum robustness and performance.

Figures (14)

• Figure 1: (Left) Illustration of NVM crossbar which produces output current $I_j$, as a dot-product of voltage vector, $V_i$ and NVM device conductance, $G_{ij}$. (Right) Various peripheral and parasitic resistances modify the dot-product computations into an interdependent function of the analog variables (voltage, conductance and resistances) in a non-ideal NVM crossbar.
• Figure 2: Digital vs Analog Natural Test Accuracy for vanilla and adversarially trained DNNs. clean: vanilla training with unperturbed images. pgd-epsN: PGD adversarial training with $\epsilon_{train}= N = [2,4,6,8]$ and iter $=50$
• Figure 3: Signal to Noise ($SNR$) at the output of every layer. for vanilla and adversarially trained DNNs. clean: vanilla training with unperturbed images. pgd-epsN: PGD adversarial training with $\epsilon_{train}= N = [2,4,6,8]$ and iter $=50$. NVM crossbar model: 64x64_100k ($NF = 0.26$).
• Figure 4: Noise Sensitivity at the output of every layer. for vanilla and adversarially trained DNNs. clean: vanilla training with unperturbed images. pgd-epsN: PGD adversarial training with $\epsilon_{train}= N = [2,4,6,8]$ and iter $=50$. NVM crossbar model: 64x64_100k ($NF = 0.26$).
• Figure 5: (a)(b) Layer Cushion $\mu_i$ of layer $i$, (c)(d) Activation Contraction $c_i$ of layer $i$, for vanilla and adversarially trained DNNs. NVM crossbar model: None.