Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

PAC-Bayesian risk bounds for fully connected deep neural network with Gaussian priors

The Tien Mai

TL;DR

This work analyzes fully connected Bayesian deep neural networks with Gaussian priors through PAC-Bayesian bounds, using an exponentially weighted aggregate to obtain prediction-risk guarantees. In both nonparametric regression and binary classification with logistic loss, the authors derive upper bounds that are minimax-optimal up to logarithmic factors for functions in Besov spaces, under 1-Lipschitz activations. A key contribution is extending PAC-Bayesian analysis to deep, dense architectures rather than sparse networks, and providing sharp oracle-type inequalities for misclassification risk. The results demonstrate that fully connected Bayesian DNNs can achieve strong generalization performance with rigorous probabilistic guarantees, motivating further study of priors and architectures in uncertainty quantification and learning theory.

Abstract

Deep neural networks (DNNs) have emerged as a powerful methodology with significant practical successes in fields such as computer vision and natural language processing. Recent works have demonstrated that sparsely connected DNNs with carefully designed architectures can achieve minimax estimation rates under classical smoothness assumptions. However, subsequent studies revealed that simple fully connected DNNs can achieve comparable convergence rates, challenging the necessity of sparsity. Theoretical advances in Bayesian neural networks (BNNs) have been more fragmented. Much of those work has concentrated on sparse networks, leaving the theoretical properties of fully connected BNNs underexplored. In this paper, we address this gap by investigating fully connected Bayesian DNNs with Gaussian prior using PAC-Bayes bounds. We establish upper bounds on the prediction risk for a probabilistic deep neural network method, showing that these bounds match (up to logarithmic factors) the minimax-optimal rates in Besov space, for both nonparametric regression and binary classification with logistic loss. Importantly, our results hold for a broad class of practical activation functions that are Lipschitz continuous.

PAC-Bayesian risk bounds for fully connected deep neural network with Gaussian priors

TL;DR

This work analyzes fully connected Bayesian deep neural networks with Gaussian priors through PAC-Bayesian bounds, using an exponentially weighted aggregate to obtain prediction-risk guarantees. In both nonparametric regression and binary classification with logistic loss, the authors derive upper bounds that are minimax-optimal up to logarithmic factors for functions in Besov spaces, under 1-Lipschitz activations. A key contribution is extending PAC-Bayesian analysis to deep, dense architectures rather than sparse networks, and providing sharp oracle-type inequalities for misclassification risk. The results demonstrate that fully connected Bayesian DNNs can achieve strong generalization performance with rigorous probabilistic guarantees, motivating further study of priors and architectures in uncertainty quantification and learning theory.

Abstract

Deep neural networks (DNNs) have emerged as a powerful methodology with significant practical successes in fields such as computer vision and natural language processing. Recent works have demonstrated that sparsely connected DNNs with carefully designed architectures can achieve minimax estimation rates under classical smoothness assumptions. However, subsequent studies revealed that simple fully connected DNNs can achieve comparable convergence rates, challenging the necessity of sparsity. Theoretical advances in Bayesian neural networks (BNNs) have been more fragmented. Much of those work has concentrated on sparse networks, leaving the theoretical properties of fully connected BNNs underexplored. In this paper, we address this gap by investigating fully connected Bayesian DNNs with Gaussian prior using PAC-Bayes bounds. We establish upper bounds on the prediction risk for a probabilistic deep neural network method, showing that these bounds match (up to logarithmic factors) the minimax-optimal rates in Besov space, for both nonparametric regression and binary classification with logistic loss. Importantly, our results hold for a broad class of practical activation functions that are Lipschitz continuous.
Paper Structure (15 sections, 9 theorems, 36 equations)

This paper contains 15 sections, 9 theorems, 36 equations.

Table of Contents

  1. Introduction
  2. Structure of the paper
  3. Notations and definitions
  4. Notations
  5. Fully connected deep neural networks
  6. Nonparametric regression
  7. Problem setting
  8. Minimax-optimal excess risk bound in the Besov space
  9. Results for binary classification
  10. Setting
  11. Excess risk bounds for cross-entropy loss
  12. Misclassification excess risk bounds
  13. Conclusion
  14. Proof
  15. Auxiliary lemmas

Key Result

Theorem 1

Assume that Assumption asm1 and assu_bounded are satisfied. Put $C^*_{\sigma, \varsigma} := 16[C^2+\sigma^2+ C(\varsigma \lor(2C) ]$ and $\lambda = n/ C^*_{\sigma, \varsigma}$. We consider the architectures as in eq_langer_architecture, then and with any $\delta \in (0,1)$ that: where $\psi_n := n^{-\frac{-2\beta}{2\beta +d}} (\log n)^3$ and $\mathcal{C } >0$ is some universal constants dependi

Theorems & Definitions (17)

  • Theorem 1
  • Remark 1
  • Corollary 1
  • Remark 2
  • Remark 3
  • Remark 4
  • Proposition 1
  • Theorem 2
  • Remark 5
  • Corollary 2
  • ...and 7 more