Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Provable Generalization Bounds for Deep Neural Networks with Momentum-Adaptive Gradient Dropout

Adeel Safder

TL;DR

This work addresses overfitting in overparameterized deep neural networks by introducing MAGDrop, a momentum-adaptive dropout that modulates activation-level sparsity based on current gradients and momentum from optimizers. The authors derive a non-asymptotic, computable PAC-Bayes generalization bound with explicit constants that accounts for MAGDrop's adaptivity, showing up to $29.2\%$ tighter bounds relative to static regularization. Empirically, activation-based MAGDrop achieves competitive test accuracies on MNIST and CIFAR-10 with small generalization gaps, and is demonstrated on Tiny ImageNet to explore scaling behavior, all while providing fully reproducible code and numerical bound calculations. The work bridges theory and practice by delivering a concrete regularization mechanism with provable generalization guarantees, offering a robust framework for improving DNN generalization in high-stakes applications.

Abstract

Deep neural networks (DNNs) achieve remarkable performance but often suffer from overfitting due to their high capacity. We introduce Momentum-Adaptive Gradient Dropout (MAGDrop), a novel regularization method that dynamically adjusts dropout rates on activations based on current gradients and accumulated momentum, enhancing stability in non-convex optimization landscapes. To theoretically justify MAGDrop's effectiveness, we derive a non-asymptotic, computable PAC-Bayes generalization bound that accounts for its adaptive nature, achieving up to 29.2\% tighter bounds compared to standard approaches by leveraging momentum-driven perturbation control. Empirically, the activation-based MAGDrop achieves competitive performance on MNIST (99.52\%) and CIFAR-10 (92.03\%), with generalization gaps of 0.48\% and 6.52\%, respectively. We provide fully reproducible code and numerical computation of our bounds to validate our theoretical claims. Our work bridges theoretical insights and practical advancements, offering a robust framework for enhancing DNN generalization, making it suitable for high-stakes applications.

Provable Generalization Bounds for Deep Neural Networks with Momentum-Adaptive Gradient Dropout

TL;DR

This work addresses overfitting in overparameterized deep neural networks by introducing MAGDrop, a momentum-adaptive dropout that modulates activation-level sparsity based on current gradients and momentum from optimizers. The authors derive a non-asymptotic, computable PAC-Bayes generalization bound with explicit constants that accounts for MAGDrop's adaptivity, showing up to tighter bounds relative to static regularization. Empirically, activation-based MAGDrop achieves competitive test accuracies on MNIST and CIFAR-10 with small generalization gaps, and is demonstrated on Tiny ImageNet to explore scaling behavior, all while providing fully reproducible code and numerical bound calculations. The work bridges theory and practice by delivering a concrete regularization mechanism with provable generalization guarantees, offering a robust framework for improving DNN generalization in high-stakes applications.

Abstract

Deep neural networks (DNNs) achieve remarkable performance but often suffer from overfitting due to their high capacity. We introduce Momentum-Adaptive Gradient Dropout (MAGDrop), a novel regularization method that dynamically adjusts dropout rates on activations based on current gradients and accumulated momentum, enhancing stability in non-convex optimization landscapes. To theoretically justify MAGDrop's effectiveness, we derive a non-asymptotic, computable PAC-Bayes generalization bound that accounts for its adaptive nature, achieving up to 29.2\% tighter bounds compared to standard approaches by leveraging momentum-driven perturbation control. Empirically, the activation-based MAGDrop achieves competitive performance on MNIST (99.52\%) and CIFAR-10 (92.03\%), with generalization gaps of 0.48\% and 6.52\%, respectively. We provide fully reproducible code and numerical computation of our bounds to validate our theoretical claims. Our work bridges theoretical insights and practical advancements, offering a robust framework for enhancing DNN generalization, making it suitable for high-stakes applications.
Paper Structure (22 sections, 1 theorem, 9 equations, 3 tables, 1 algorithm)

This paper contains 22 sections, 1 theorem, 9 equations, 3 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Related Work
  3. Generalization Bounds.
  4. Regularization Techniques.
  5. Gaps and Our Contribution.
  6. Momentum-Adaptive Gradient Dropout (MAGDrop)
  7. MAGDrop Algorithm
  8. Implementation and Hyperparameters
  9. Theoretical Analysis
  10. Non-Asymptotic PAC-Bayes Bound
  11. Experiments
  12. Tiny ImageNet Results
  13. Computational Limitations and Future Work
  14. Reproducibility
  15. Discussion
  16. ...and 7 more sections

Key Result

Theorem 1

For a DNN with MAGDrop, dataset $S$ of size $m$, bounded loss $\ell \leq B$, inputs $\|x\| \leq X$, and spectrally normalized weights $\|W_l\|_2 \leq \kappa_l$, with probability at least $1-\delta$, the generalization error is bounded by: where $p_{t,l}$ is the adaptive dropout rate on activations, $\mathbb{E}[p_{t,l}] \leq p_{\text{base}} / (1 + \beta)$, $\alpha = 0.5$ is a constant from the ent

Theorems & Definitions (2)

  • Theorem 1
  • proof