Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Position: Many generalization measures for deep learning are fragile

Shuofeng Zhang, Ard Louis

TL;DR

The paper argues that many post-mortem generalization measures for deep neural networks are fragile to modest training changes, risking misleading interpretations of generalization. It conducts a fragility audit across training-hyperparameters, temporal dynamics after interpolation, and data-complexity changes, showing that measures like path, spectral, Frobenius norms, flatness proxies, and certain PAC-Bayes surrogates can invert trends or inflate without changing test accuracy. By contrast, a function-space Marginal-Likelihood PAC-Bayes bound tracks data complexity and learning-curve scaling more robustly, highlighting the value of function-space diagnostics. The authors propose practical fragility metrics (CMS/eCMS) and a symmetry-based Exp++ protocol to stress-test measures, call for reparameterization-invariant, data-aware diagnostics, and suggest building a fragility benchmark to improve evaluation practice in deep learning generalization research.

Abstract

A wide variety of generalization measures have been applied to deep neural networks (DNNs). Although obtaining tight bounds remains challenging, such measures are often assumed to reproduce qualitative generalization trends. In this position paper, we argue that many post-mortem generalization measures -- those computed on trained networks -- are \textbf{fragile}: small training modifications that barely affect the underlying DNN can substantially change a measure's value, trend, or scaling behavior. For example, minor hyperparameter changes, such as learning rate adjustments or switching between SGD variants can reverse the slope of a learning curve in widely used generalization measures like the path norm. We also identify subtler forms of fragility. For instance, the PAC-Bayes origin measure is regarded as one of the most reliable, and is indeed less sensitive to hyperparameter tweaks than many other measures. However, it completely fails to capture differences in data complexity across learning curves. This data fragility contrasts with the function-based marginal-likelihood PAC-Bayes bound, which does capture differences in data-complexity, including scaling behavior, in learning curves, but which is not a post-mortem measure. Beyond demonstrating that many bounds -- such as path, spectral and Frobenius norms, flatness proxies, and deterministic PAC-Bayes surrogates -- are fragile, this position paper also argues that developers of new measures should explicitly audit them for fragility.

Position: Many generalization measures for deep learning are fragile

TL;DR

The paper argues that many post-mortem generalization measures for deep neural networks are fragile to modest training changes, risking misleading interpretations of generalization. It conducts a fragility audit across training-hyperparameters, temporal dynamics after interpolation, and data-complexity changes, showing that measures like path, spectral, Frobenius norms, flatness proxies, and certain PAC-Bayes surrogates can invert trends or inflate without changing test accuracy. By contrast, a function-space Marginal-Likelihood PAC-Bayes bound tracks data complexity and learning-curve scaling more robustly, highlighting the value of function-space diagnostics. The authors propose practical fragility metrics (CMS/eCMS) and a symmetry-based Exp++ protocol to stress-test measures, call for reparameterization-invariant, data-aware diagnostics, and suggest building a fragility benchmark to improve evaluation practice in deep learning generalization research.

Abstract

A wide variety of generalization measures have been applied to deep neural networks (DNNs). Although obtaining tight bounds remains challenging, such measures are often assumed to reproduce qualitative generalization trends. In this position paper, we argue that many post-mortem generalization measures -- those computed on trained networks -- are \textbf{fragile}: small training modifications that barely affect the underlying DNN can substantially change a measure's value, trend, or scaling behavior. For example, minor hyperparameter changes, such as learning rate adjustments or switching between SGD variants can reverse the slope of a learning curve in widely used generalization measures like the path norm. We also identify subtler forms of fragility. For instance, the PAC-Bayes origin measure is regarded as one of the most reliable, and is indeed less sensitive to hyperparameter tweaks than many other measures. However, it completely fails to capture differences in data complexity across learning curves. This data fragility contrasts with the function-based marginal-likelihood PAC-Bayes bound, which does capture differences in data-complexity, including scaling behavior, in learning curves, but which is not a post-mortem measure. Beyond demonstrating that many bounds -- such as path, spectral and Frobenius norms, flatness proxies, and deterministic PAC-Bayes surrogates -- are fragile, this position paper also argues that developers of new measures should explicitly audit them for fragility.
Paper Structure (52 sections, 4 theorems, 25 equations, 23 figures, 5 tables, 2 algorithms)

This paper contains 52 sections, 4 theorems, 25 equations, 23 figures, 5 tables, 2 algorithms.

Table of Contents

  1. Introduction
  2. Related work
  3. From classical capacity to modern practice.
  4. Capacity-oriented diagnostics: norms, margins, distance from initialization
  5. Geometry-oriented diagnostics: flatness and sharpness
  6. Algorithm-aware certificates: PAC-Bayes as operational measures
  7. Other perspectives.
  8. Meta-evaluations and synthesis.
  9. Training-hyperparameter fragility
  10. Temporal behavior fragility
  11. Data-Complexity Fragility: Label Noise & Dataset Difficulty
  12. Scale-invariant network and exponential learning rate schedule
  13. Quantitative fragility: a scale‑free instability score
  14. Setup.
  15. Randomness adjustment.
  16. ...and 37 more sections

Key Result

Theorem 6.2

(GD+WD+LR fixed $\Leftrightarrow$ GD+WD$\searrow$ + LR$\nearrow$). Let $f(\theta_t)$ be scale-invariant and let training use SGD with momentum $\gamma$. Introduce the shorthands Define the interval endpoints and set Consider the two updates (with $\theta_{-1}=\theta_0$, $\tilde{\theta}_0=\theta_0$, $\tilde{\theta}_{-1}=\alpha\,\theta_{-1}$): If $\alpha\in\mathcal{I}$, then the schedules ensur

Figures (23)

  • Figure 1: Path‑norm learning curves under small training‑pipeline changes (no early stopping).Left:SGD with momentum, LR $=0.01$. Middle:Adam, LR $=0.01$. Right:Adam, LR $=0.001$. Curves show ResNet‑50 test error (black), path norm (blue), and path norm over margin (orange); axes are logarithmic and error bars denote seed variability. Training set sizes $n$ range from $10^2$ to the full FashionMNIST training set ($5\times 10^4$ samples).
  • Figure 2: ResNet‑50 on FashionMNIST as a function of training epochs. Left (SGD+momentum, LR $=0.01$). Middle (Adam, LR $=0.01$). Right (Adam, LR $=0.001$). While the path norm and path norm over margin measures show qualitative different behavior upon small hyperparameter tweaks, the underlying generalization is similar across the three panels.
  • Figure 3: Label‑corruption sweeps with post‑mortem norms vs. function‑space ML‑PACBayes on RESNET-50.Left:Adam, $\mathrm{LR}{=}0.01$. Middle:Adam, $\mathrm{LR}{=}0.001$. Right: ML‑PACBayes; purple, Dashed black: test error; blue: path norm; orange: path norm over margin. A ten‑fold LR change reverses the path‑norm trend (left vs. middle), while ML‑PACBayes remains a smooth, monotone function of corruption (right) and is, by construction, agnostic to optimizer/LR. All panels use 10,000 training samples on binarized FashionMNIST.
  • Figure 4: Fixed architecture, varying dataset.. The model is RESNET-50. Left:PACBAYES_ORIG with Adam. Middle:PACBAYES_ORIG with SGD (momentum). Changing from ADAM to SGD with momentum leads to a small increase in the overall scale of the PACBAYES_ORIG learning curves (solid lines). However, for both optimisers, these learning curves don't distinguish MNIST, FashionMNIST, and CIFAR10, missing both the data‑complexity ordering and the changing slopes visible in the underlying learning curves (dashed lines). Right: ML‑PACBayes (marginal‑likelihood PAC–Bayes) is tight and preserves the correct dataset ordering and reproduces the trends in the slope; for technical reasons, this panel is computed on the binarized versions of the three datasets.
  • Figure S1: Frobenius distance, ResNet-50 on CIFAR-10 (Adam). Left: $\eta{=}10^{-3}$ shows a plateau and late collapse; right: $\eta{=}10^{-2}$ decays smoothly while accuracy tracks similarly.
  • ...and 18 more figures

Theorems & Definitions (16)

  • Definition 6.1: Scale invariant neural networks
  • Theorem 6.2: Equivalence of schedules in scale-invariant nets
  • Remark 6.3: Parameter range and nonnegativity
  • Definition 7.1: Conditional Measure Spread (CMS)
  • Definition 7.2: eCMS (Excess CMS)
  • Lemma 3.1: Scale-invariant networks
  • proof
  • Definition 3.2: SGD with momentum and WD
  • Definition 3.3: Equivalent states
  • Lemma 3.4: Equivalent scaling commutes with GD
  • ...and 6 more