Enhancing Neural Network Robustness Against Fault Injection Through Non-linear Weight Transformations
Ninnart Fuengfusin, Hakaru Tamukoh
TL;DR
This work tackles hardware-induced fault tolerance in neural networks by constraining weights through Saturated Activation Functions (SAFs) during training, so that at inference time faulty weights are mitigated by applying SAFs. The method formalizes weight transformation a_i = σ(τ(W_i) a_{i-1} + b_i) and deployment-time processing a_i = σ(τ(Ĵ_i) a_{i-1} + b_i) with Ŵ_i = f(W_i, BER), using SAFs such as Tanh, Arctan, Softsign, and a Modified Tanh. It demonstrates across CIFAR-10/100 and ImageNet-2012, for FP32, FP16, and Q2.5, that fault injections at BER = $10^{-5}$ cause far smaller accuracy losses when SAFs are applied, and that leveraging pre-trained weights for SAF adaptation can accelerate training. The results indicate substantial improvements in bit-flip resilience with minimal computational overhead, offering a practical path to robust deployment of large DNNs on fault-prone hardware.
Abstract
Deploying deep neural networks (DNNs) in real-world environments poses challenges due to faults that can manifest in physical hardware from radiation, aging, and temperature fluctuations. To address this, previous works have focused on protecting DNNs via activation range restriction using clipped ReLU and finding the optimal clipping threshold. However, this work instead focuses on constraining DNN weights by applying saturated activation functions (SAFs): Tanh, Arctan, and others. SAFs prevent faults from causing DNN weights to become excessively large, which can lead to model failure. These methods not only enhance the robustness of DNNs against fault injections but also improve DNN performance by a small margin. Before deployment, DNNs are trained with weights constrained by SAFs. During deployment, the weights without applied SAF are written to mediums with faults. When read, weights with faults are applied with SAFs and are used for inference. We demonstrate our proposed method across three datasets (CIFAR10, CIFAR100, ImageNet 2012) and across three datatypes (32-bit floating point (FP32), 16-bit floating point, and 8-bit fixed point). We show that our method enables FP32 ResNet18 with ImageNet 2012 to operate at a bit-error rate of 0.00001 with minor accuracy loss, while without the proposed method, the FP32 DNN only produces random guesses. Furthermore, to accelerate the training process, we demonstrate that an ImageNet 2012 pre-trained ResNet18 can be adapted to SAF by training for a few epochs with a slight improvement in Top-1 accuracy while still ensuring robustness against fault injection.