Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Improving Analog Neural Network Robustness: A Noise-Agnostic Approach with Explainable Regularizations

Alice Duque, Pedro Freire, Egor Manuylovich, Dmitrii Stoliarov, Jaroslaw Prilepsky, Sergei Turitsyn

TL;DR

A comprehensive, hardware-agnostic solution to address both correlated and uncorrelated noise affecting the activation layers of deep neural models and introduces a new explainable regularization framework that harnesses these mechanisms to significantly enhance noise robustness in deep neural architectures.

Abstract

This work tackles the critical challenge of mitigating "hardware noise" in deep analog neural networks, a major obstacle in advancing analog signal processing devices. We propose a comprehensive, hardware-agnostic solution to address both correlated and uncorrelated noise affecting the activation layers of deep neural models. The novelty of our approach lies in its ability to demystify the "black box" nature of noise-resilient networks by revealing the underlying mechanisms that reduce sensitivity to noise. In doing so, we introduce a new explainable regularization framework that harnesses these mechanisms to significantly enhance noise robustness in deep neural architectures.

Improving Analog Neural Network Robustness: A Noise-Agnostic Approach with Explainable Regularizations

TL;DR

A comprehensive, hardware-agnostic solution to address both correlated and uncorrelated noise affecting the activation layers of deep neural models and introduces a new explainable regularization framework that harnesses these mechanisms to significantly enhance noise robustness in deep neural architectures.

Abstract

This work tackles the critical challenge of mitigating "hardware noise" in deep analog neural networks, a major obstacle in advancing analog signal processing devices. We propose a comprehensive, hardware-agnostic solution to address both correlated and uncorrelated noise affecting the activation layers of deep neural models. The novelty of our approach lies in its ability to demystify the "black box" nature of noise-resilient networks by revealing the underlying mechanisms that reduce sensitivity to noise. In doing so, we introduce a new explainable regularization framework that harnesses these mechanisms to significantly enhance noise robustness in deep neural architectures.
Paper Structure (5 sections, 7 equations, 4 figures)

This paper contains 5 sections, 7 equations, 4 figures.

Table of Contents

  1. Introduction
  2. Noise in Neural Network Models
  3. Designing Explainable Regularizers
  4. Results and discussion
  5. Conclusion

Figures (4)

  • Figure 1: Feed-forward network under (a) correlated and (b) uncorrelated noise. (c) Examples of such noise sources in analog NNs.
  • Figure 2: Per-row visualization of each weight matrix in a network with (a) standard training and (b) proposed regularization method. Regularization manages to push rows means toward zero to meet correlated noise mitigation strategy.
  • Figure 3: (a) PDF of random variable $\hat{z} \sim \mathcal{N}(\mu,0.2)$ after undergoing sigmoid transformation, for different values of $\mu$. (b) Pre-activations, (c) post-activations and (d) output weights distributions for networks with different training methods. Blue: network with standard training; Red: network with noise-aware training; Green: network with proposed regularization method.
  • Figure 4: Performance comparison between models with standard training, noise-aware training and the proposed regularized model, for MNIST and Fashion MNIST datasets.