Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Loss Gradient Gaussian Width based Generalization and Optimization Guarantees

Arindam Banerjee, Qiaobo Li, Yingxue Zhou

TL;DR

The paper shifts the focus from predictor-based uniform convergence to gradient geometry by introducing Loss Gradient Gaussian Width (LGGW) as a central complexity measure. It develops generalization guarantees under a gradient domination framework that includes the PL condition and provides optimization guarantees for gradient descent with sample reuse, showing that small LGGW keeps empirical gradients aligned with population gradients over many passes. For deep networks, the authors bound single-sample LGGW by the Gaussian width of the featurizer, tying gradient complexity to architectural choices like layer widths. The results rely on generic chaining and the Majorizing Measure Theorem to relate LGGW to vector Rademacher complexities and Gaussian processes, yielding potentially quantitative improvements over traditional Rademacher-based bounds in high-capacity models. Overall, LGGW offers a principled, geometry-aware path toward tighter generalization and optimization guarantees in deep learning, with explicit implications for FFNs and ResNets.

Abstract

Generalization and optimization guarantees on the population loss often rely on uniform convergence based analysis, typically based on the Rademacher complexity of the predictors. The rich representation power of modern models has led to concerns about this approach. In this paper, we present generalization and optimization guarantees in terms of the complexity of the gradients, as measured by the Loss Gradient Gaussian Width (LGGW). First, we introduce generalization guarantees directly in terms of the LGGW under a flexible gradient domination condition, which includes the popular PL (Polyak-Łojasiewicz) condition as a special case. Second, we show that sample reuse in iterative gradient descent does not make the empirical gradients deviate from the population gradients as long as the LGGW is small. Third, focusing on deep networks, we bound their single-sample LGGW in terms of the Gaussian width of the featurizer, i.e., the output of the last-but-one layer. To our knowledge, our generalization and optimization guarantees in terms of LGGW are the first results of its kind, and hold considerable promise towards quantitatively tight bounds for deep models.

Loss Gradient Gaussian Width based Generalization and Optimization Guarantees

TL;DR

The paper shifts the focus from predictor-based uniform convergence to gradient geometry by introducing Loss Gradient Gaussian Width (LGGW) as a central complexity measure. It develops generalization guarantees under a gradient domination framework that includes the PL condition and provides optimization guarantees for gradient descent with sample reuse, showing that small LGGW keeps empirical gradients aligned with population gradients over many passes. For deep networks, the authors bound single-sample LGGW by the Gaussian width of the featurizer, tying gradient complexity to architectural choices like layer widths. The results rely on generic chaining and the Majorizing Measure Theorem to relate LGGW to vector Rademacher complexities and Gaussian processes, yielding potentially quantitative improvements over traditional Rademacher-based bounds in high-capacity models. Overall, LGGW offers a principled, geometry-aware path toward tighter generalization and optimization guarantees in deep learning, with explicit implications for FFNs and ResNets.

Abstract

Generalization and optimization guarantees on the population loss often rely on uniform convergence based analysis, typically based on the Rademacher complexity of the predictors. The rich representation power of modern models has led to concerns about this approach. In this paper, we present generalization and optimization guarantees in terms of the complexity of the gradients, as measured by the Loss Gradient Gaussian Width (LGGW). First, we introduce generalization guarantees directly in terms of the LGGW under a flexible gradient domination condition, which includes the popular PL (Polyak-Łojasiewicz) condition as a special case. Second, we show that sample reuse in iterative gradient descent does not make the empirical gradients deviate from the population gradients as long as the LGGW is small. Third, focusing on deep networks, we bound their single-sample LGGW in terms of the Gaussian width of the featurizer, i.e., the output of the last-but-one layer. To our knowledge, our generalization and optimization guarantees in terms of LGGW are the first results of its kind, and hold considerable promise towards quantitatively tight bounds for deep models.
Paper Structure (22 sections, 30 theorems, 181 equations, 3 figures)

This paper contains 22 sections, 30 theorems, 181 equations, 3 figures.

Table of Contents

  1. Introduction
  2. Gaussian Width based Generalization Bounds
  3. Bounds based on Vector Rademacher Complexity
  4. Main Result: Gaussian Width Bound
  5. Proof sketch of Main Result
  6. Gradient Descent with Sample Reuse
  7. Gaussian Width of Deep Learning Gradients for Neural Networks
  8. Bounds Based on Empirical Geometry of Gradients
  9. Gaussian Width Bounds for Neural Networks
  10. Bounds for FeedForward Networks (FFNs)
  11. Bounds for Residual Networks (ResNets)
  12. Conclusions
  13. Acknowledgements.
  14. Generalization Bounds: Proofs for Section \ref{['sec:gen']}
  15. Generalization bound in terms of NERC
  16. ...and 7 more sections

Key Result

Proposition 1

Under Assumptions asmp:gd_cond and asmp:gradbnd, with $\theta^{\star} \in \mathop{\mathrm{argmin}}\nolimits _{\theta \in \Theta} {\cal L}_{\mathcal{D}}(\theta)$ denoting any population loss minimizer and $\hat{R}_n(\hat{\Xi}^{(n)})$ as in Definition defn:nerc, for any $\delta > 0$, with probability

Figures (3)

  • Figure 1: Gradient Domination (GD) Ratio for $\alpha=1, 2$ for (a) ResNet18 on CIFAR-10, (b) FFN on CIFAR-10, and (c) CNN on Fashion-MNIST. Each experiment is repeated 5 times and we report the average and the maximum ratio. The results show that GD holds for $\alpha=1$ with a small constant $\bar{c}_{1} < 5$ for all models, and for $\alpha=2$, with $\bar{c}_{2} < 1$ for some models, but needs large constants $\bar{c}_{2} > 100$ for for ResNet18.
  • Figure 2: (a-b) Sorted gradient coordinate in linear scale (a) and log scale (b) for SGD over 100 epochs on CIFAR-10. Model: 4-hidden-layer ReLU network with 256 nodes on each layer, with around 1,055,000 parameters. (c-d) Sorted gradient coordinate in linear scale (c) and log scale (d) for SGD over 100 epochs on MNIST. Model: 2-layer ReLU with 128 nodes on each layer, with roughly 120,000 parameters. Y-axis is the absolute value of gradient coordinates, X-axis is sorted in increasing order.
  • Figure 3: $L_{0}$ norm and $L_{1}$ norm of featurizers for (a) ResNet18 on CIFAR-10, (b) FFN on CIFAR-10, and (c) CNN on Fashion-MNIST of the last 10% epochs close to convergence. We plot both the average and maximum norms among 5 repetitions for each experiment. The figures demonstrate that for these models, as approaching convergence, the featurizers has relatively low $L_{0}$ norm and $L_{1}$ norm, which further indicates a small LGGW.

Theorems & Definitions (55)

  • Definition 1: Gradient Sets
  • Definition 2: Loss Gradient Gaussian Width
  • Definition 3
  • Remark 2.1
  • Remark 2.2
  • Proposition 1
  • Theorem 1
  • Remark 2.3
  • Remark 2.4
  • Remark 2.5
  • ...and 45 more