Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Admissibility of Stein Shrinkage for Batch Normalization in the Presence of Adversarial Attacks

Sofia Ivolgina, P. Thomas Fletcher, Baba C. Vemuri

TL;DR

This work introduces a Stein/James–Stein shrinkage approach to batch normalization (BN) statistics, proving that shrinkage estimators dominate standard BN estimators under adversarial perturbations modeled as sub-Gaussian noise. The authors derive JS-based corrections for both BN means and BN variances (the latter via a Gamma-distribution framework) and prove dominance theorems in Gaussian and Gamma contexts. They further demonstrate substantial robustness improvements across CIFAR-10, Cityscapes, and PPMI data, including resilience to FGSM/PGD attacks, by yielding more stable BN statistics and smoother loss landscapes. The combination of theoretical guarantees and extensive experiments suggests practical benefits for improving robustness of deep networks without sacrificing accuracy, particularly in small-batch or adversarial settings.

Abstract

Batch normalization (BN) is a ubiquitous operation in deep neural networks, primarily used to improve stability and regularization during training. BN centers and scales feature maps using sample means and variances, which are naturally suited for Stein's shrinkage estimation. Applying such shrinkage yields more accurate mean and variance estimates of the batch in the mean-squared-error sense. In this paper, we prove that the Stein shrinkage estimator of the mean and variance dominates over the sample mean and variance estimators, respectively, in the presence of adversarial attacks modeled using sub-Gaussian distributions. Furthermore, by construction, the James-Stein (JS) BN yields a smaller local Lipschitz constant compared to the vanilla BN, implying better regularity properties and potentially improved robustness. This facilitates and justifies the application of Stein shrinkage to estimate the mean and variance parameters in BN and the use of it in image classification and segmentation tasks with and without adversarial attacks. We present SOTA performance results using this Stein-corrected BN in a standard ResNet architecture applied to the task of image classification using CIFAR-10 data, 3D CNN on PPMI (neuroimaging) data, and image segmentation using HRNet on Cityscape data with and without adversarial attacks.

Admissibility of Stein Shrinkage for Batch Normalization in the Presence of Adversarial Attacks

TL;DR

This work introduces a Stein/James–Stein shrinkage approach to batch normalization (BN) statistics, proving that shrinkage estimators dominate standard BN estimators under adversarial perturbations modeled as sub-Gaussian noise. The authors derive JS-based corrections for both BN means and BN variances (the latter via a Gamma-distribution framework) and prove dominance theorems in Gaussian and Gamma contexts. They further demonstrate substantial robustness improvements across CIFAR-10, Cityscapes, and PPMI data, including resilience to FGSM/PGD attacks, by yielding more stable BN statistics and smoother loss landscapes. The combination of theoretical guarantees and extensive experiments suggests practical benefits for improving robustness of deep networks without sacrificing accuracy, particularly in small-batch or adversarial settings.

Abstract

Batch normalization (BN) is a ubiquitous operation in deep neural networks, primarily used to improve stability and regularization during training. BN centers and scales feature maps using sample means and variances, which are naturally suited for Stein's shrinkage estimation. Applying such shrinkage yields more accurate mean and variance estimates of the batch in the mean-squared-error sense. In this paper, we prove that the Stein shrinkage estimator of the mean and variance dominates over the sample mean and variance estimators, respectively, in the presence of adversarial attacks modeled using sub-Gaussian distributions. Furthermore, by construction, the James-Stein (JS) BN yields a smaller local Lipschitz constant compared to the vanilla BN, implying better regularity properties and potentially improved robustness. This facilitates and justifies the application of Stein shrinkage to estimate the mean and variance parameters in BN and the use of it in image classification and segmentation tasks with and without adversarial attacks. We present SOTA performance results using this Stein-corrected BN in a standard ResNet architecture applied to the task of image classification using CIFAR-10 data, 3D CNN on PPMI (neuroimaging) data, and image segmentation using HRNet on Cityscape data with and without adversarial attacks.

Paper Structure

This paper contains 19 sections, 5 theorems, 105 equations, 7 tables.

Table of Contents

  1. Introduction
  2. A Brief Note on the JS-shrinkage Estimator
  3. Relevance of JS-estimator to BN
  4. Stein's Shrinkage Applied to BN
  5. Dominance of Stein Shrinkage in the Presence of Adversarial Attacks
  6. Dominance of the JS-estimator under Sub-Gaussian Perturbations for Normally Distributed Random Variables
  7. Dominance of the JS-estimator under Sub-Gaussian Perturbations for Gamma Distributed Random Variables
  8. Experiments
  9. Application to Cifar10 Data
  10. Application to Cityscapes Data
  11. Application to PPMI Data
  12. Experiments using Data Dependent Attacks
  13. Discussion and Conclusion
  14. Technical Appendices and Supplementary Material
  15. Experiments
  16. ...and 4 more sections

Key Result

Theorem 1

Let $\theta \in \mathbb{R}^p$ be an unknown parameter vector. Suppose that we observe $Z = X + Y$, where $X \sim \mathcal{N}_p(\theta, I_p)$ and $Y \in \mathbb{R}^p$ is an independent $Y \sim \mathrm{SG}(2\varepsilon^2)$. Define the JS-estimator by and the observation-based estimator by $\hat{\theta}^{0} = Z$. Then, for all $\theta \in \mathbb{R}^p$ and for all $p \geq 3$ the JS-estimator strictl

Theorems & Definitions (6)

  • Theorem 1
  • Theorem 2: Dominance of the JS-Estimator for the Gamma Model
  • Theorem 3
  • Remark 4
  • Lemma 5: Stein's Lemma for the Gamma Distribution
  • Lemma 6: Probability bound for $\hat{\sigma}^2_{y,i} + W_i$