Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Accuracy-Robustness Trade Off via Spiking Neural Network Gradient Sparsity Trail

Luu Trong Nhan, Luu Trung Duong, Pham Ngoc Nam, Truong Cong Thang

TL;DR

The paper investigates adversarial robustness in Spiking Neural Networks (SNNs) and uncovers a natural gradient sparsity phenomenon that can yield state-of-the-art robustness without explicit defenses under certain architectures. Through empirical benchmarking on CIFAR and event-based datasets, it reveals a robustness-generalization trade-off driven by gradient density, with architectural choices like pooling shaping gradient sparsity. The authors provide theoretical bounds linking input-gradient sparsity to reduced weight-gradient density, and show that reducing gradient sparsity (e.g., via average pooling) can improve clean generalization at the cost of robustness. These findings offer a principled perspective on designing sparse-gradient SNNs for robust, energy-efficient vision systems and suggest future sparsity-aware architectural strategies.

Abstract

Spiking Neural Networks (SNNs) have attracted growing interest in both computational neuroscience and artificial intelligence, primarily due to their inherent energy efficiency and compact memory footprint. However, achieving adversarial robustness in SNNs, (particularly for vision-related tasks) remains a nascent and underexplored challenge. Recent studies have proposed leveraging sparse gradients as a form of regularization to enhance robustness against adversarial perturbations. In this work, we present a surprising finding: under specific architectural configurations, SNNs exhibit natural gradient sparsity and can achieve state-of-the-art adversarial defense performance without the need for any explicit regularization. Further analysis reveals a trade-off between robustness and generalization: while sparse gradients contribute to improved adversarial resilience, they can impair the model's ability to generalize; conversely, denser gradients support better generalization but increase vulnerability to attacks. Our findings offer new insights into the dual role of gradient sparsity in SNN training.

Accuracy-Robustness Trade Off via Spiking Neural Network Gradient Sparsity Trail

TL;DR

The paper investigates adversarial robustness in Spiking Neural Networks (SNNs) and uncovers a natural gradient sparsity phenomenon that can yield state-of-the-art robustness without explicit defenses under certain architectures. Through empirical benchmarking on CIFAR and event-based datasets, it reveals a robustness-generalization trade-off driven by gradient density, with architectural choices like pooling shaping gradient sparsity. The authors provide theoretical bounds linking input-gradient sparsity to reduced weight-gradient density, and show that reducing gradient sparsity (e.g., via average pooling) can improve clean generalization at the cost of robustness. These findings offer a principled perspective on designing sparse-gradient SNNs for robust, energy-efficient vision systems and suggest future sparsity-aware architectural strategies.

Abstract

Spiking Neural Networks (SNNs) have attracted growing interest in both computational neuroscience and artificial intelligence, primarily due to their inherent energy efficiency and compact memory footprint. However, achieving adversarial robustness in SNNs, (particularly for vision-related tasks) remains a nascent and underexplored challenge. Recent studies have proposed leveraging sparse gradients as a form of regularization to enhance robustness against adversarial perturbations. In this work, we present a surprising finding: under specific architectural configurations, SNNs exhibit natural gradient sparsity and can achieve state-of-the-art adversarial defense performance without the need for any explicit regularization. Further analysis reveals a trade-off between robustness and generalization: while sparse gradients contribute to improved adversarial resilience, they can impair the model's ability to generalize; conversely, denser gradients support better generalization but increase vulnerability to attacks. Our findings offer new insights into the dual role of gradient sparsity in SNN training.

Paper Structure

This paper contains 19 sections, 8 theorems, 61 equations, 6 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related works
  3. Spiking Neural Networks
  4. Adversarial Robustness In Artificial Neural Network
  5. Adversarial Robustness in Spiking Neural Network
  6. Adversarial attacks
  7. Fast Gradient Sign Method (FGSM)
  8. Fast Gradient Sign Method (FGSM)
  9. Projected Gradient Descent (PGD)
  10. Evaluation and analysis
  11. Overall settings
  12. Natural adversarial robustness of SNN over temporal-coded backpropagation
  13. Gradient analysis and theoretical explanation
  14. Gradient analysis
  15. Theoretical explanation
  16. ...and 4 more sections

Key Result

Theorem 4.1

Suppose $f$ is a differentiable SNN by surrogate gradients, and $\epsilon$ is the magnitude of an attack, assumed to be small enough. Given an input image $\mathbf{x}$ with corresponding label $y$, the ratio of adversarial vulnerability $\rho_{\text{adv}}(f, \mathbf{x}, \epsilon, \ell_{\infty})$ and where $\rho_{\text{adv}}(f, \mathbf{x}, \epsilon, \ell_{\infty})$ scale linearly with non-zeros ent

Figures (6)

  • Figure 1: Example illustration of perturbated images from CIFAR-10 with FGSM and PGD over a network.
  • Figure 2: Effect of adversarial attacks over multiple perturbation magnitude $\epsilon$ between defended WideResNet16 and vanilla SEW-ResNet18.
  • Figure 3: Inspection of average input gradient sparsity and input gradient norm across all employed operations within SEW-ResNet18. Gradient was clearly defined and high sparsity mostly occurs in operations of SEW residual blocks.
  • Figure 4: Architecture of SEW residual block proposed by fang2021deep
  • Figure 5: Inspection of average input gradient sparsity and input gradient norm across all employed operations within SEW-ResNet18 when employed with average pooling. Sparsity significantly reduced when compared with result in Figure \ref{['sparse_fig_sew']}.
  • ...and 1 more figures

Theorems & Definitions (16)

  • Theorem 4.1: liu2024enhancing
  • Theorem 4.2: luu2024improvement
  • Theorem 4.3
  • proof
  • Theorem 4.4
  • proof
  • Definition A.1: Random Vulnerability
  • Definition A.2: Adversarial Vulnerability
  • Theorem A.1: Liu et al.
  • proof
  • ...and 6 more