Adversarial Training from Mean Field Perspective

TL;DR

This work provides a theoretical framework based on mean-field theory to analyze adversarial training in random deep networks without data-distribution assumptions. It introduces a linear-like, two-Gaussian representation that captures the probabilistic properties of the entire network and the input–parameter dependence, enabling derivation of upper bounds on adversarial loss for multiple norm combinations. The results reveal key insights: vanilla networks struggle to be adversarially trainable, residual networks maintain trainability, adversarial training regularizes weights and degrades capacity (with width mitigating degradation), and the bounds depend on network depth and dimensional factors in a nuanced way. Empirical experiments corroborate the theoretical predictions in early training stages and illustrate the framework’s potential applicability to other training paradigms beyond adversarial training. Overall, the TEXT mean-field approach advances the theoretical understanding of adversarial robustness and provides a versatile tool for studying deep learning dynamics under perturbations.

Abstract

Although adversarial training is known to be effective against adversarial examples, training dynamics are not well understood. In this study, we present the first theoretical analysis of adversarial training in random deep neural networks without any assumptions on data distributions. We introduce a new theoretical framework based on mean field theory, which addresses the limitations of existing mean field-based approaches. Based on this framework, we derive (empirically tight) upper bounds of $\ell_q$ norm-based adversarial loss with $\ell_p$ norm-based adversarial examples for various values of $p$ and $q$. Moreover, we prove that networks without shortcuts are generally not adversarially trainable and that adversarial training reduces network capacity. We also show that network width alleviates these issues. Furthermore, we present the various impacts of the input and output dimensions on the upper bounds and time evolution of the weight variance.

Figures (22)

• Figure 1: Distribution of $J(\bm{x}^\mathrm{in})_{1,1}$ in the vanilla ReLU network with $d=1,000$, $K=1$, $N=5,000$, $L=10$, $\sigma^2_w=2$, and $\sigma^2_b=0.01$. The blue histogram represents the experimental results (10,000-time samplings), and the orange curve is predicted by \ref{['th:MFT']}.
• Figure 2: Values of $\beta_{p,q}$. Under further assumptions, we can obtain equality of the adversarial loss rather than inequality (upper bound). Values marked with $\dag$ represent the equality when $\epsilon$ is sufficiently small. Values marked with $\diamondsuit$ are applicable if $\epsilon$ is sufficiently small and $K=1$.
• Figure 3: Adversarial loss (\ref{['eq:adv-loss-def']}) in vanilla networks with $N=40,000$, $K=100$, $L=3$, and $\epsilon=0.1$. We generated 100 adversarial examples for each input dimension. The blue curves and bands represent the mean and standard deviation of the adversarial loss, respectively, whereas the orange curves (upper bounds) are predicted based on \ref{['th:bounds']}. Some samples slightly exceed the upper bounds because we used the finite network width (cf. \ref{['sec:other-experiments']}).
• Figure 4: Time evolution of the weight variance in the vanilla network with $L=10$, $p=\infty$, $q=\infty$, and $\epsilon=0.3$. We used $N=1,000$ for standard and $\ell_2$ regularized training. The solid lines represent experimental results. The dashed lines are predicted by \ref{['th:evolution-vanilla']}.
• Figure 5: Heat map of the training accuracy of vanilla networks with $p=\infty$, $q=\infty$, and $\epsilon=0.3$. The dashed lines represent the condition of $T$ in \ref{['th:trainability-vanilla']} with $m=0.0001$. In standard training, high accuracy is obtained across all the depths and widths (cf. \ref{['fig:trainability']}).

Theorems & Definitions (79)

• Definition 3.1: ReLU-like network
• Theorem 4.1: Properties and distributions of $\J(\bxin)$ and $\a(\bxin)$
• Theorem 5.1: Upper bounds of adversarial loss
• Theorem 5.4: Weight time evolution of vanilla network in adversarial training
• Definition 5.5: $(M,m)$-trainability condition
• Lemma 5.5: Vanilla and residual $(M,m)$-trainability condition
• Theorem 5.6: Vanilla networks are not adversarially trainable
• Theorem 5.7: Residual networks are adversarially trainable
• Theorem 5.8: Adversarial training degrades network capacity
• Proposition D.1