Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Approximation and Generalization Abilities of Score-based Neural Network Generative Models for Sub-Gaussian Distributions

Guoji Fu, Wee Sun Lee

TL;DR

This work provides a theoretically rigorous framework for score-based neural network generative models (SGMs) targeting sub-Gaussian data. It develops a universal approach using regularized KDE-based score estimators and compact ReLU neural networks to approximate and learn scores, obtaining near-minimax rates in total variation for distribution estimation, without requiring Lipschitz score or density lower bounds. Early stopping is leveraged to transfer convergence guarantees from smoothed densities to Sobolev and Besov densities, yielding nearly optimal rates up to logs. The results broaden the theoretical foundations of SGMs and suggest principled design choices for score estimation and neural network architectures in high-dimensional settings.

Abstract

This paper studies the approximation and generalization abilities of score-based neural network generative models (SGMs) in estimating an unknown distribution $P_0$ from $n$ i.i.d. observations in $d$ dimensions. Assuming merely that $P_0$ is $α$-sub-Gaussian, we prove that for any time step $t \in [t_0, n^{\mathcal{O}(1)}]$, where $t_0 > \mathcal{O}(α^2n^{-2/d}\log n)$, there exists a deep ReLU neural network with width $\leq \mathcal{O}(n^{\frac{3}{d}}\log_2n)$ and depth $\leq \mathcal{O}(\log^2n)$ that can approximate the scores with $\tilde{\mathcal{O}}(n^{-1})$ mean square error and achieve a nearly optimal rate of $\tilde{\mathcal{O}}(n^{-1}t_0^{-d/2})$ for score estimation, as measured by the score matching loss. Our framework is universal and can be used to establish convergence rates for SGMs under milder assumptions than previous work. For example, assuming further that the target density function $p_0$ lies in Sobolev or Besov classes, with an appropriately early stopping strategy, we demonstrate that neural network-based SGMs can attain nearly minimax convergence rates up to logarithmic factors. Our analysis removes several crucial assumptions, such as Lipschitz continuity of the score function or a strictly positive lower bound on the target density.

Approximation and Generalization Abilities of Score-based Neural Network Generative Models for Sub-Gaussian Distributions

TL;DR

This work provides a theoretically rigorous framework for score-based neural network generative models (SGMs) targeting sub-Gaussian data. It develops a universal approach using regularized KDE-based score estimators and compact ReLU neural networks to approximate and learn scores, obtaining near-minimax rates in total variation for distribution estimation, without requiring Lipschitz score or density lower bounds. Early stopping is leveraged to transfer convergence guarantees from smoothed densities to Sobolev and Besov densities, yielding nearly optimal rates up to logs. The results broaden the theoretical foundations of SGMs and suggest principled design choices for score estimation and neural network architectures in high-dimensional settings.

Abstract

This paper studies the approximation and generalization abilities of score-based neural network generative models (SGMs) in estimating an unknown distribution from i.i.d. observations in dimensions. Assuming merely that is -sub-Gaussian, we prove that for any time step , where , there exists a deep ReLU neural network with width and depth that can approximate the scores with mean square error and achieve a nearly optimal rate of for score estimation, as measured by the score matching loss. Our framework is universal and can be used to establish convergence rates for SGMs under milder assumptions than previous work. For example, assuming further that the target density function lies in Sobolev or Besov classes, with an appropriately early stopping strategy, we demonstrate that neural network-based SGMs can attain nearly minimax convergence rates up to logarithmic factors. Our analysis removes several crucial assumptions, such as Lipschitz continuity of the score function or a strictly positive lower bound on the target density.
Paper Structure (61 sections, 61 theorems, 580 equations, 1 figure, 1 table)

This paper contains 61 sections, 61 theorems, 580 equations, 1 figure, 1 table.

Table of Contents

  1. Introduction
  2. Our contributions
  3. Preliminaries and Background
  4. Notations and definitions
  5. Score-based generative models
  6. Main Results
  7. Assumptions
  8. Score estimation by regularized empirical score functions
  9. Score approximation and estimation by deep neural networks
  10. Neural network score approximation
  11. Neural network score estimation
  12. Distribution estimation errors of SGMs for sub-Gaussian distributions
  13. Proof Sketch
  14. Proof sketch of \ref{['thrm:sub-Gau-score-est']}
  15. Proof sketch of \ref{['thrm:approx:score-match-sub-Gau-nn']}
  16. ...and 46 more sections

Key Result

Lemma 1

For any $d \geq 1, n \geq 3$, Let $P$ be a $\alpha$-sub-Gaussian distribution on ${\mathbb{R}}^d$ and $\hat{P}$ be its empirical distribution associated to a sample $\{{\bm{x}}^{(i)}\}_{i=1}^n$. For any $\sigma \gtrsim \alpha n^{-1/d}\log^{1/2}n$, let $P_{\sigma} = P * {\mathcal{N}}(0, \sigma^2{\bm{

Figures (1)

  • Figure 1: An illustration of the network architecture implementing $\tilde{\phi}$ for approximating $x^2$ on $[a, b]$.

Theorems & Definitions (112)

  • Definition 1: Sub-Gaussian Distribution vershynin2018high
  • Lemma 1
  • Remark 1
  • Theorem 1: Neural Network Score Approximation for Sub-Gaussian Distributions
  • Theorem 2: Neural Network Score Estimation for Sub-Gaussian Distributions
  • Theorem 3: Distribution Estimation Error of $P_{t_0}$
  • Corollary 1: Distribution Estimation Error for Sobolev Class of Density
  • Corollary 2: Distribution Estimation Error for Besov Class of Density
  • Remark 2: Dependence of Network Width on Dimension $d$ and the Bias-Variance Trade-Off
  • Remark 3
  • ...and 102 more