Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Error Bounds of Supervised Classification from Information-Theoretic Perspective

Binchuan Qi

TL;DR

An upper bound on fitting error is derived by utilizing the triangle inequality, which is applied to provide theoretical explanations for overparameterization, non-convex optimization and flat minima in deep learning.

Abstract

In this paper, we explore bounds on the expected risk when using deep neural networks for supervised classification from an information theoretic perspective. Firstly, we introduce model risk and fitting error, which are derived from further decomposing the empirical risk. Model risk represents the expected value of the loss under the model's predicted probabilities and is exclusively dependent on the model. Fitting error measures the disparity between the empirical risk and model risk. Then, we derive the upper bound on fitting error, which links the back-propagated gradient and the model's parameter count with the fitting error. Furthermore, we demonstrate that the generalization errors are bounded by the classification uncertainty, which is characterized by both the smoothness of the distribution and the sample size. Based on the bounds on fitting error and generalization, by utilizing the triangle inequality, we establish an upper bound on the expected risk. This bound is applied to provide theoretical explanations for overparameterization, non-convex optimization and flat minima in deep learning. Finally, empirical verification confirms a significant positive correlation between the derived theoretical bounds and the practical expected risk, thereby affirming the practical relevance of the theoretical findings.

Error Bounds of Supervised Classification from Information-Theoretic Perspective

TL;DR

An upper bound on fitting error is derived by utilizing the triangle inequality, which is applied to provide theoretical explanations for overparameterization, non-convex optimization and flat minima in deep learning.

Abstract

In this paper, we explore bounds on the expected risk when using deep neural networks for supervised classification from an information theoretic perspective. Firstly, we introduce model risk and fitting error, which are derived from further decomposing the empirical risk. Model risk represents the expected value of the loss under the model's predicted probabilities and is exclusively dependent on the model. Fitting error measures the disparity between the empirical risk and model risk. Then, we derive the upper bound on fitting error, which links the back-propagated gradient and the model's parameter count with the fitting error. Furthermore, we demonstrate that the generalization errors are bounded by the classification uncertainty, which is characterized by both the smoothness of the distribution and the sample size. Based on the bounds on fitting error and generalization, by utilizing the triangle inequality, we establish an upper bound on the expected risk. This bound is applied to provide theoretical explanations for overparameterization, non-convex optimization and flat minima in deep learning. Finally, empirical verification confirms a significant positive correlation between the derived theoretical bounds and the practical expected risk, thereby affirming the practical relevance of the theoretical findings.
Paper Structure (12 sections, 14 theorems, 77 equations, 3 figures, 1 table)

This paper contains 12 sections, 14 theorems, 77 equations, 3 figures, 1 table.

Table of Contents

  1. Introduction
  2. Related Work
  3. Preliminaries
  4. Bound on Fitting Error
  5. Bound on Generalization Error
  6. Bound on Expected Risk
  7. Application to Deep Learning
  8. Overparameterization and Fitting Error
  9. Non-convex optimization
  10. Relationship between the Bound on Fitting Error and Flat Minima theory
  11. Empirical Validations
  12. Conclusion

Key Result

Lemma 1

Let $Z$ be a non-negative random variable. Then, for all $t \geq 0$, $\operatorname{Pr}(Z \geq t) \leq \mathbb{E}[Z] / t$.

Figures (3)

  • Figure 1: Model architectures and configuration parameters.
  • Figure 2: Model architectures and configuration parameters.
  • Figure 3: Model architectures and configuration parameters.

Theorems & Definitions (31)

  • Definition 1: Model risk
  • Definition 2: Fitting error
  • Definition 3: normalized fitting error
  • Definition 4: Structural matrix
  • Lemma 1: Markov inequality
  • Lemma 2
  • proof
  • Lemma 3
  • proof
  • Theorem 1: Bound on fitting error
  • ...and 21 more