Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Enhancing Reliability of Neural Networks at the Edge: Inverted Normalization with Stochastic Affine Transformations

Soyed Tuhin Ahmed, Kamal Danouchi, Guillaume Prenat, Lorena Anghel, Mehdi B. Tahoori

TL;DR

This work tackles the reliability gap of Bayesian neural networks deployed on in-memory computing hardware using non-volatile memories. It introduces inverted normalization, where the affine transformation precedes normalization, and stochastic affine transformations via affine Dropout to inherently absorb IMC non-idealities. By treating affine parameters as learnable and applying Monte Carlo inference, the method maintains uncertainty estimation while achieving substantial accuracy and fault-tolerance improvements, up to 58.11% over conventional baselines and 78.95% OOD detection effectiveness. The approach is demonstrated across diverse tasks and bit-precision levels, showing generalizability and practical impact for safe, energy-efficient edge AI.

Abstract

Bayesian Neural Networks (BayNNs) naturally provide uncertainty in their predictions, making them a suitable choice in safety-critical applications. Additionally, their realization using memristor-based in-memory computing (IMC) architectures enables them for resource-constrained edge applications. In addition to predictive uncertainty, however, the ability to be inherently robust to noise in computation is also essential to ensure functional safety. In particular, memristor-based IMCs are susceptible to various sources of non-idealities such as manufacturing and runtime variations, drift, and failure, which can significantly reduce inference accuracy. In this paper, we propose a method to inherently enhance the robustness and inference accuracy of BayNNs deployed in IMC architectures. To achieve this, we introduce a novel normalization layer combined with stochastic affine transformations. Empirical results in various benchmark datasets show a graceful degradation in inference accuracy, with an improvement of up to $58.11\%$.

Enhancing Reliability of Neural Networks at the Edge: Inverted Normalization with Stochastic Affine Transformations

TL;DR

This work tackles the reliability gap of Bayesian neural networks deployed on in-memory computing hardware using non-volatile memories. It introduces inverted normalization, where the affine transformation precedes normalization, and stochastic affine transformations via affine Dropout to inherently absorb IMC non-idealities. By treating affine parameters as learnable and applying Monte Carlo inference, the method maintains uncertainty estimation while achieving substantial accuracy and fault-tolerance improvements, up to 58.11% over conventional baselines and 78.95% OOD detection effectiveness. The approach is demonstrated across diverse tasks and bit-precision levels, showing generalizability and practical impact for safe, energy-efficient edge AI.

Abstract

Bayesian Neural Networks (BayNNs) naturally provide uncertainty in their predictions, making them a suitable choice in safety-critical applications. Additionally, their realization using memristor-based in-memory computing (IMC) architectures enables them for resource-constrained edge applications. In addition to predictive uncertainty, however, the ability to be inherently robust to noise in computation is also essential to ensure functional safety. In particular, memristor-based IMCs are susceptible to various sources of non-idealities such as manufacturing and runtime variations, drift, and failure, which can significantly reduce inference accuracy. In this paper, we propose a method to inherently enhance the robustness and inference accuracy of BayNNs deployed in IMC architectures. To achieve this, we introduce a novel normalization layer combined with stochastic affine transformations. Empirical results in various benchmark datasets show a graceful degradation in inference accuracy, with an improvement of up to .
Paper Structure (22 sections, 2 equations, 7 figures, 1 table)

This paper contains 22 sections, 2 equations, 7 figures, 1 table.

Table of Contents

  1. Introduction
  2. Background
  3. Bayesian Neural Networks
  4. Normalization Methods in Deep Learning
  5. Dropout
  6. In-Memory Computation using NVMs
  7. Related Works
  8. Methodology
  9. Inverted Normalization
  10. Affine Dropout
  11. Initialization of Affine Parameters
  12. Bayesian Inference and Uncertainty Estimation
  13. Results
  14. Simulation Setup
  15. Evaluated Models and Training Settings
  16. ...and 7 more sections

Figures (7)

  • Figure 1: Change in activation distribution due to faults.
  • Figure 2: Computation flow of the proposed and conventional normalization layers.
  • Figure 3: Operation flow for the proposed affine parameters (weight and biases) Dropout.
  • Figure 4: Examples of non-idealities: (a) Stochastic switching in magnetic memories under different voltages and (b) influence of temperature on the resistance distributions (Monte Carlo simulations).
  • Figure 5: Evaluation of robustness of ResNet-18 and U-Net topologies on CIFAR-10 and DRIVE datasets. The shaded region shows $\pm$ one standard deviation variation from the mean. The left and right figures of both datasets illustrate the evaluation of bit-flips and additive conductance variations, respectively.
  • ...and 2 more figures