Hessian-based Analysis of Large Batch Training and Robustness to Adversaries

TL;DR

This work analyzes how large-batch training alters the loss landscape through the Hessian spectrum and its implications for robustness. By computing the true Hessian spectrum and examining perturbations along dominant eigenvectors, it shows large batches land in higher-curvature regions, not saddle points, contributing to generalization gaps. The study further connects robustness to optimization, proving the adversarial perturbation problem is saddle-free almost everywhere under typical DNN assumptions and demonstrating that robust training pushes toward flatter minima with smaller Hessian spectra. These insights suggest that integrating robust optimization can mitigate large-batch drawbacks, with practical implications validated across multiple architectures and datasets.

Abstract

Large batch size training of Neural Networks has been shown to incur accuracy loss when trained with the current methods. The exact underlying reasons for this are still not completely understood. Here, we study large batch size training through the lens of the Hessian operator and robust optimization. In particular, we perform a Hessian based study to analyze exactly how the landscape of the loss function changes when training with large batch size. We compute the true Hessian spectrum, without approximation, by back-propagating the second derivative. Extensive experiments on multiple networks show that saddle-points are not the cause for generalization gap of large batch size training, and the results consistently show that large batch converges to points with noticeably higher Hessian spectrum. Furthermore, we show that robust training allows one to favor flat areas, as points with large Hessian spectrum show poor robustness to adversarial perturbation. We further study this relationship, and provide empirical and theoretical proof that the inner loop for robust training is a saddle-free optimization problem \textit{almost everywhere}. We present detailed experiments with five different network architectures, including a residual network, tested on MNIST, CIFAR-10, and CIFAR-100 datasets. We have open sourced our method which can be accessed at [1].

Figures (16)

• Figure 1: Top 20 eigenvalues of the Hessian is shown for C1 on CIFAR-10 (left) and M1 on MNIST (right) datasets. The spectrum is computed using power iteration with relative error of $1\text{\sc{e}-}{4}$.
• Figure 2: The landscape of the loss is shown along the dominant eigenvector, $v_1$, of the Hessian for C1 on CIFAR-10 dataset. Here $\epsilon$ is a scalar that perturbs the model parameters along $v_1$.
• Figure 3: The landscape of the loss is shown when the C1 model parameters are changed along the first two dominant eigenvectors of the Hessian with the perturbation magnitude $\epsilon_1$ and $\epsilon_2$.
• Figure 4: Changes in the dominant eigenvalue of the Hessian w.r.t weights and the total gradient is shown for different epochs during training. Note the increase in $\lambda_1^\theta$ (blue curve) for large batch v.s. small batch. In particular, note that the values for total gradient along with the Hessian spectrum show that large batch does not get "stuck" in saddle points, but areas in the optimization landscape that have high curvature. More results are shown in Fig. \ref{['fig:qalex_h_logger_appendix']}. The dotted points show the corresponding results when using robust optimization, which makes the solver stay in areas with smaller spectrum.
• Figure 5: 1-D Parametric plot for C3 model on CIFAR-10. We interpolate between parameters of $\mathcal{M}_{ORI}$ and $\mathcal{M}_{ADV}$, and compute the cross entropy loss on the y-axis.

Theorems & Definitions (4)

• Theorem 1
• Proposition 2
• Lemma 3
• proof