Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Concentration Inequalities for Stochastic Optimization of Unbounded Objective Functions with Application to Denoising Score Matching

Jeremiah Birrell

TL;DR

This work develops concentration inequalities for stochastic optimization with unbounded objective functions and unbounded support by generalizing McDiarmid's inequality to sample-dependent, locally Lipschitz bounds and by deriving a distribution-dependent Rademacher complexity bound for unbounded function classes. It establishes a uniform law of large numbers with sample reuse when data are paired with easily-sampled auxiliary variables, and then applies these results to denoising score matching (DSM) and generative adversarial networks (GANs), quantifying the benefits of sample reuse. The key contributions include explicit $L^1$ and high-probability bounds for stochastic optima, accommodating heavy-tailed distributions and unbounded objectives, with practical implications for training DSM and GANs using auxiliary Gaussian inputs. Overall, the framework broadens the repertoire of provable guarantees in stochastic optimization under realistic, unbounded-tail settings and clarifies the impact of reusing auxiliary randomness in learning algorithms.

Abstract

We derive novel concentration inequalities that bound the statistical error for a large class of stochastic optimization problems, focusing on the case of unbounded objective functions. Our derivations utilize the following key tools: 1) A new form of McDiarmid's inequality that is based on sample-dependent one-component mean-difference bounds and which leads to a novel uniform law of large numbers result for unbounded functions. 2) A new Rademacher complexity bound for families of functions that satisfy an appropriate sample-dependent Lipschitz property, which allows for application to a large class of distributions with unbounded support. As an application of these results, we derive statistical error bounds for denoising score matching (DSM), an application that inherently requires one to consider unbounded objective functions and distributions with unbounded support, even in cases where the data distribution has bounded support. In addition, our results quantify the benefit of sample-reuse in algorithms that employ easily-sampled auxiliary random variables in addition to the training data, e.g., as in DSM, which uses auxiliary Gaussian random variables.

Concentration Inequalities for Stochastic Optimization of Unbounded Objective Functions with Application to Denoising Score Matching

TL;DR

This work develops concentration inequalities for stochastic optimization with unbounded objective functions and unbounded support by generalizing McDiarmid's inequality to sample-dependent, locally Lipschitz bounds and by deriving a distribution-dependent Rademacher complexity bound for unbounded function classes. It establishes a uniform law of large numbers with sample reuse when data are paired with easily-sampled auxiliary variables, and then applies these results to denoising score matching (DSM) and generative adversarial networks (GANs), quantifying the benefits of sample reuse. The key contributions include explicit and high-probability bounds for stochastic optima, accommodating heavy-tailed distributions and unbounded objectives, with practical implications for training DSM and GANs using auxiliary Gaussian inputs. Overall, the framework broadens the repertoire of provable guarantees in stochastic optimization under realistic, unbounded-tail settings and clarifies the impact of reusing auxiliary randomness in learning algorithms.

Abstract

We derive novel concentration inequalities that bound the statistical error for a large class of stochastic optimization problems, focusing on the case of unbounded objective functions. Our derivations utilize the following key tools: 1) A new form of McDiarmid's inequality that is based on sample-dependent one-component mean-difference bounds and which leads to a novel uniform law of large numbers result for unbounded functions. 2) A new Rademacher complexity bound for families of functions that satisfy an appropriate sample-dependent Lipschitz property, which allows for application to a large class of distributions with unbounded support. As an application of these results, we derive statistical error bounds for denoising score matching (DSM), an application that inherently requires one to consider unbounded objective functions and distributions with unbounded support, even in cases where the data distribution has bounded support. In addition, our results quantify the benefit of sample-reuse in algorithms that employ easily-sampled auxiliary random variables in addition to the training data, e.g., as in DSM, which uses auxiliary Gaussian random variables.

Paper Structure

This paper contains 19 sections, 20 theorems, 130 equations, 1 table.

Table of Contents

  1. Introduction
  2. Contributions
  3. Related Works
  4. A Generalization of McDiarmid's Inequality for Unbounded Functions
  5. Use of Orlicz Norms
  6. Comparison of Theorem \ref{['thm:McDiarmid_general_mean_bound_method']} with McDiarmid's Inequality and Related Methods
  7. A Uniform Law of Large Numbers Concentration Inequality for Unbounded Functions with Sample Reuse
  8. Bounding the Mean
  9. Combining the Concentration Inequality and Mean Bound
  10. Concentration Inequalities for Stochastic Optimization Problems with Sample Reuse
  11. Applications
  12. Denoising Score Matching
  13. Generative Adversarial Networks
  14. Conclusion
  15. Generalization of Theorem \ref{['thm:McDiarmid_general_mean_bound_method']}
  16. ...and 4 more sections

Key Result

Theorem 2.1

Let $(\mathcal{X}_i,P_i)$, $i=1,...,n$ be probability spaces, define $\mathcal{X}^n\coloneqq\prod_{i=1}^n \mathcal{X}_i$ with the product $\sigma$-algebra, and let $P^n\coloneqq\prod_{i=1}^n P_i$. Suppose $\phi:\mathcal{X}^n\to\mathbb{R}$ satisfies the following: Then we have the MGF bound and for all $t\geq 0$ we have the concentration inequality

Theorems & Definitions (50)

  • Theorem 2.1
  • Remark 2.2
  • proof
  • Lemma 2.3
  • Corollary 2.4
  • Corollary 2.5
  • Remark 3.2
  • Lemma 3.3
  • Remark 3.4
  • proof
  • ...and 40 more