Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Non-vacuous Generalization Bounds for Deep Neural Networks without any modification to the trained models

Khoat Than, Dat Phan

TL;DR

This work tackles the challenge of explaining deep neural network generalization with non-vacuous bounds for a fixed, pretrained model without modifying the network. It introduces two principal, non-modification bounds and a data-augmentation bound, together with a concentration inequality for conditionally independent variables to support the theory. The tractable bound is computable from the training set using a partition of the input space, while a distribution-aware bound reveals how data geometry influences generalization; a data-augmentation bound enables meaningful model comparisons when training losses are similar. Empirically, the bounds are evaluated on 32 ImageNet-pretrained models and are shown to be non-vacuous at large scale, providing practical certificates and enabling model comparison through augmentation.

Abstract

Deep neural network (NN) with millions or billions of parameters can perform really well on unseen data, after being trained from a finite training set. Various prior theories have been developed to explain such excellent ability of NNs, but do not provide a meaningful bound on the test error. Some recent theories, based on PAC-Bayes and mutual information, are non-vacuous and hence show a great potential to explain the excellent performance of NNs. However, they often require a stringent assumption and extensive modification (e.g. compression, quantization) to the trained model of interest. Therefore, those prior theories provide a guarantee for the modified versions only. In this paper, we propose two novel bounds on the test error of a model. Our bounds uses the training set only and require no modification to the model. Those bounds are verified on a large class of modern NNs, pretrained by Pytorch on the ImageNet dataset, and are non-vacuous. To the best of our knowledge, these are the first non-vacuous bounds at this large scale, without any modification to the pretrained models.

Non-vacuous Generalization Bounds for Deep Neural Networks without any modification to the trained models

TL;DR

This work tackles the challenge of explaining deep neural network generalization with non-vacuous bounds for a fixed, pretrained model without modifying the network. It introduces two principal, non-modification bounds and a data-augmentation bound, together with a concentration inequality for conditionally independent variables to support the theory. The tractable bound is computable from the training set using a partition of the input space, while a distribution-aware bound reveals how data geometry influences generalization; a data-augmentation bound enables meaningful model comparisons when training losses are similar. Empirically, the bounds are evaluated on 32 ImageNet-pretrained models and are shown to be non-vacuous at large scale, providing practical certificates and enabling model comparison through augmentation.

Abstract

Deep neural network (NN) with millions or billions of parameters can perform really well on unseen data, after being trained from a finite training set. Various prior theories have been developed to explain such excellent ability of NNs, but do not provide a meaningful bound on the test error. Some recent theories, based on PAC-Bayes and mutual information, are non-vacuous and hence show a great potential to explain the excellent performance of NNs. However, they often require a stringent assumption and extensive modification (e.g. compression, quantization) to the trained model of interest. Therefore, those prior theories provide a guarantee for the modified versions only. In this paper, we propose two novel bounds on the test error of a model. Our bounds uses the training set only and require no modification to the model. Those bounds are verified on a large class of modern NNs, pretrained by Pytorch on the ImageNet dataset, and are non-vacuous. To the best of our knowledge, these are the first non-vacuous bounds at this large scale, without any modification to the pretrained models.

Paper Structure

This paper contains 16 sections, 16 theorems, 43 equations, 3 figures, 3 tables.

Table of Contents

  1. Introduction
  2. Related work
  3. Error bounds
  4. Tractable bounds
  5. General bound
  6. Concentration inequality for conditionally independent variables
  7. Empirical evaluation
  8. Large-scale evaluation for pretrained models
  9. Evaluation with data augmentation
  10. Conclusion
  11. Proofs for main results
  12. Supporting theorems and lemmas
  13. Hoeffding's inequality for conditionals
  14. Hoeffding's Lemma for conditionals
  15. Small random variables
  16. ...and 1 more sections

Key Result

Theorem 1

Consider a model ${\bm{h}}$, a dataset ${\bm{S}}$ with $n$ i.i.d. samples from distribution $P$, and a partition $\Gamma$. For any constants $\gamma \ge 1, \delta >0$ and $\alpha \in [0, \frac{\gamma n(K+\gamma n)}{K(4n-3)}]$, we have the following with probability at least $1- \gamma^{-\alpha} -\d where $\hat{u} = \frac{\gamma}{2n} + \frac{\gamma^2 }{2}\sum_{i=1}^K \left( \frac{n_i}{n} \right)^

Figures (3)

  • Figure 1: The dynamic of $\hat{n} = \sum_{i=1}^K \left( \frac{n_i}{n} \right)^2$ as $K$ changes.
  • Figure 2: The uncertainty $\text{Unc}(\Gamma) = C\sqrt{\hat{u}\alpha \ln\gamma } + g(\delta/2)$ as (right) $K$ changes and (left) $\alpha$ changes, for fixed $K = 200, \gamma = 0.04^{-1/\alpha}, \delta = 0.01$.
  • Figure 3: The dynamic of bound (\ref{['thm-gen-train-small-K-any-distribution-aug-eq']}) as the noise level $\sigma$ increases. These subfigures report the main part $\bar{\epsilon}({\bm{h}}) + F(\hat{{\bm{S}}},{\bm{h}})$ of the bound.

Theorems & Definitions (30)

  • Theorem 1
  • proof : Proof sketch
  • Corollary 1
  • Theorem 2
  • Theorem 3
  • Theorem 4
  • Lemma 1
  • proof : Proof of Theorem \ref{['thm-gen-train-small-K-any-distribution']}
  • proof : Proof of Theorem \ref{['thm-gen-train-small-K-any-distribution-aug']}
  • proof : Proof of Theorem \ref{['thm-gen-train-small-K']}
  • ...and 20 more