Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

On Hardening DNNs against Noisy Computations

Xiao Wang, Hendrik Borras, Bernhard Klein, Holger Fröning

TL;DR

The paper tackles maintaining DNN accuracy when executing on noisy analog hardware. It compares quantization-aware training (QAT) with constant and dynamic scaling against traditional noisy training, using CIFAR-10 across LeNet-5, VGG-11, and ResNet-18 and measuring robustness with the midpoint noise level $\mu$. It finds that noisy training frequently yields stronger noise tolerance than QAT, especially for deeper architectures, though constant-scaling QAT can improve robustness in noise-free training. The results highlight the challenge of error amplification in deep networks and suggest that combining robustness techniques with architectural design is essential for practical deployment on noisy analog accelerators.

Abstract

The success of deep learning has sparked significant interest in designing computer hardware optimized for the high computational demands of neural network inference. As further miniaturization of digital CMOS processors becomes increasingly challenging, alternative computing paradigms, such as analog computing, are gaining consideration. Particularly for compute-intensive tasks such as matrix multiplication, analog computing presents a promising alternative due to its potential for significantly higher energy efficiency compared to conventional digital technology. However, analog computations are inherently noisy, which makes it challenging to maintain high accuracy on deep neural networks. This work investigates the effectiveness of training neural networks with quantization to increase the robustness against noise. Experimental results across various network architectures show that quantization-aware training with constant scaling factors enhances robustness. We compare these methods with noisy training, which incorporates a noise injection during training that mimics the noise encountered during inference. While both two methods increase tolerance against noise, noisy training emerges as the superior approach for achieving robust neural network performance, especially in complex neural architectures.

On Hardening DNNs against Noisy Computations

TL;DR

The paper tackles maintaining DNN accuracy when executing on noisy analog hardware. It compares quantization-aware training (QAT) with constant and dynamic scaling against traditional noisy training, using CIFAR-10 across LeNet-5, VGG-11, and ResNet-18 and measuring robustness with the midpoint noise level . It finds that noisy training frequently yields stronger noise tolerance than QAT, especially for deeper architectures, though constant-scaling QAT can improve robustness in noise-free training. The results highlight the challenge of error amplification in deep networks and suggest that combining robustness techniques with architectural design is essential for practical deployment on noisy analog accelerators.

Abstract

The success of deep learning has sparked significant interest in designing computer hardware optimized for the high computational demands of neural network inference. As further miniaturization of digital CMOS processors becomes increasingly challenging, alternative computing paradigms, such as analog computing, are gaining consideration. Particularly for compute-intensive tasks such as matrix multiplication, analog computing presents a promising alternative due to its potential for significantly higher energy efficiency compared to conventional digital technology. However, analog computations are inherently noisy, which makes it challenging to maintain high accuracy on deep neural networks. This work investigates the effectiveness of training neural networks with quantization to increase the robustness against noise. Experimental results across various network architectures show that quantization-aware training with constant scaling factors enhances robustness. We compare these methods with noisy training, which incorporates a noise injection during training that mimics the noise encountered during inference. While both two methods increase tolerance against noise, noisy training emerges as the superior approach for achieving robust neural network performance, especially in complex neural architectures.
Paper Structure (12 sections, 5 equations, 3 figures, 2 tables)

This paper contains 12 sections, 5 equations, 3 figures, 2 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Methodology
  4. Robustness: midpoint noise level
  5. Quantization
  6. Experiments
  7. Robustness of QAT without noisy training
  8. Results with LeNet-5
  9. Results with VGG and ResNet
  10. Experiments with Noisy Training.
  11. Comparison among Architectures
  12. Conclusion

Figures (3)

  • Figure 1: Global noise injection in a quantized neural network.
  • Figure 2: Robustness of LeNet-5 on CIFAR-10, quantized with different bit widths, using either constant scaling (scaling factors $s$) or dynamic scaling on activations. Note that the x-axis is of logarithmic scale.
  • Figure 3: Robustness of LeNet-5 and ResNet-18 on CIFAR-10 with noisy training, combined with quantization.