Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhance DNN Adversarial Robustness and Efficiency via Injecting Noise to Non-Essential Neurons

Zhenyu Liu, Garrett Gagnon, Swagath Venkataramani, Liu Liu

TL;DR

The paper addresses the dual challenge of adversarial vulnerability and high compute cost in DNNs by introducing Non-Essential Neurons Noise Injection, a data-driven, non-uniform noise strategy. It combines a learning-based approximation with a random projection, enabling selective noise injection on non-essential neurons while preserving essential ones, and leverages structured N:M granularity to boost hardware efficiency. The approach demonstrates robust accuracy gains across ResNet-18 and WideResNet-34-10 on CIFAR-10/100, with substantial reductions in compute (BitOPs) and without retraining from scratch, outperforming several SOTA defenses in many settings. Practically, the method offers a tunable, hardware-friendly defense that can be applied to pre-trained networks, making robust and efficient deployment feasible on resource-constrained devices.

Abstract

Deep Neural Networks (DNNs) have revolutionized a wide range of industries, from healthcare and finance to automotive, by offering unparalleled capabilities in data analysis and decision-making. Despite their transforming impact, DNNs face two critical challenges: the vulnerability to adversarial attacks and the increasing computational costs associated with more complex and larger models. In this paper, we introduce an effective method designed to simultaneously enhance adversarial robustness and execution efficiency. Unlike prior studies that enhance robustness via uniformly injecting noise, we introduce a non-uniform noise injection algorithm, strategically applied at each DNN layer to disrupt adversarial perturbations introduced in attacks. By employing approximation techniques, our approach identifies and protects essential neurons while strategically introducing noise into non-essential neurons. Our experimental results demonstrate that our method successfully enhances both robustness and efficiency across several attack scenarios, model architectures, and datasets.

Enhance DNN Adversarial Robustness and Efficiency via Injecting Noise to Non-Essential Neurons

TL;DR

The paper addresses the dual challenge of adversarial vulnerability and high compute cost in DNNs by introducing Non-Essential Neurons Noise Injection, a data-driven, non-uniform noise strategy. It combines a learning-based approximation with a random projection, enabling selective noise injection on non-essential neurons while preserving essential ones, and leverages structured N:M granularity to boost hardware efficiency. The approach demonstrates robust accuracy gains across ResNet-18 and WideResNet-34-10 on CIFAR-10/100, with substantial reductions in compute (BitOPs) and without retraining from scratch, outperforming several SOTA defenses in many settings. Practically, the method offers a tunable, hardware-friendly defense that can be applied to pre-trained networks, making robust and efficient deployment feasible on resource-constrained devices.

Abstract

Deep Neural Networks (DNNs) have revolutionized a wide range of industries, from healthcare and finance to automotive, by offering unparalleled capabilities in data analysis and decision-making. Despite their transforming impact, DNNs face two critical challenges: the vulnerability to adversarial attacks and the increasing computational costs associated with more complex and larger models. In this paper, we introduce an effective method designed to simultaneously enhance adversarial robustness and execution efficiency. Unlike prior studies that enhance robustness via uniformly injecting noise, we introduce a non-uniform noise injection algorithm, strategically applied at each DNN layer to disrupt adversarial perturbations introduced in attacks. By employing approximation techniques, our approach identifies and protects essential neurons while strategically introducing noise into non-essential neurons. Our experimental results demonstrate that our method successfully enhances both robustness and efficiency across several attack scenarios, model architectures, and datasets.
Paper Structure (18 sections, 18 equations, 5 figures, 4 tables)

This paper contains 18 sections, 18 equations, 5 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work and Background
  3. Approach
  4. Learning-based Approximation Method
  5. Selection of Essential Neurons and Noise Injection
  6. Preservation of Clean Accuracy
  7. Improvement of Adversarial Robustness
  8. Implications for Efficient Execution
  9. Evaluation
  10. Evaluation Methodology
  11. Results On CIFAR-10/100
  12. Scalability to Stronger Perturbations
  13. Compare with SOTA Defense Methods
  14. Ablation Study
  15. Conclusion
  16. ...and 3 more sections

Figures (5)

  • Figure 1: Overview of Non-Essential Neuron Noise Injection: Firstly, (a) we select non-essential neurons in the low-rank space and perform noise injection on these neurons. This step is efficiently integrated into the natural process of random projection, incurring minimal additional overhead. Secondly, (b) we return to the high-dimensional space for fine-tuning. Lastly, (c) the noises introduced during training and defense are randomized, which contributes to the improvement in robustness.
  • Figure 2: Structured Non-essential Neurons Noise Injection: After getting the approximate module from fine-tuning, the Top-K algorithm is utilized to take out the index of the largest value of $N$, and then the corresponding mask $m$ is generated. Accurate module carries out the N:M Sparsity through the mask $m$, and the final result is still a mixture of approximate and accurate modules.
  • Figure 3: Evaluation stronger PGD attacks using different $\epsilon$ and steps. (ResNet-18 on CIFAR-10 and CIFAR-100)
  • Figure 4: Variations in clean and robust accuracy (PGD$^{20}$) relative to the proportion of Noise Injection.
  • Figure 5: Normalized value of various metrics across different levels of perturbation generated from our hardware simulator. NOTE: sixteen BitOps = four 4-bit additions = one 4-bit multiplication.