Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Uniform Laws of Large Numbers in Product Spaces

Ron Holzman, Shay Moran, Alexander Shlimovich

Abstract

Uniform laws of large numbers form a cornerstone of Vapnik--Chervonenkis theory, where they are characterized by the finiteness of the VC dimension. In this work, we study uniform convergence phenomena in cartesian product spaces, under assumptions on the underlying distribution that are compatible with the product structure. Specifically, we assume that the distribution is absolutely continuous with respect to the product of its marginals, a condition that captures many natural settings, including product distributions, sparse mixtures of product distributions, distributions with low mutual information, and more. We show that, under this assumption, a uniform law of large numbers holds for a family of events if and only if the linear VC dimension of the family is finite. The linear VC dimension is defined as the maximum size of a shattered set that lies on an axis-parallel line, namely, a set of vectors that agree on all but at most one coordinate. This dimension is always at most the classical VC dimension, yet it can be arbitrarily smaller. For instance, the family of convex sets in $\mathbb{R}^d$ has linear VC dimension $2$, while its VC dimension is infinite already for $d\ge 2$. Our proofs rely on estimator that departs substantially from the standard empirical mean estimator and exhibits more intricate structure. We show that such deviations from the standard empirical mean estimator are unavoidable in this setting. Throughout the paper, we propose several open questions, with a particular focus on quantitative sample complexity bounds.

Uniform Laws of Large Numbers in Product Spaces

Abstract

Uniform laws of large numbers form a cornerstone of Vapnik--Chervonenkis theory, where they are characterized by the finiteness of the VC dimension. In this work, we study uniform convergence phenomena in cartesian product spaces, under assumptions on the underlying distribution that are compatible with the product structure. Specifically, we assume that the distribution is absolutely continuous with respect to the product of its marginals, a condition that captures many natural settings, including product distributions, sparse mixtures of product distributions, distributions with low mutual information, and more. We show that, under this assumption, a uniform law of large numbers holds for a family of events if and only if the linear VC dimension of the family is finite. The linear VC dimension is defined as the maximum size of a shattered set that lies on an axis-parallel line, namely, a set of vectors that agree on all but at most one coordinate. This dimension is always at most the classical VC dimension, yet it can be arbitrarily smaller. For instance, the family of convex sets in has linear VC dimension , while its VC dimension is infinite already for . Our proofs rely on estimator that departs substantially from the standard empirical mean estimator and exhibits more intricate structure. We show that such deviations from the standard empirical mean estimator are unavoidable in this setting. Throughout the paper, we propose several open questions, with a particular focus on quantitative sample complexity bounds.

Paper Structure

This paper contains 44 sections, 30 theorems, 255 equations, 1 figure.

Table of Contents

  1. Introduction
  2. Uniform Estimation
  3. Roadmap.
  4. Main Result
  5. Uniform Box--Continuity.
  6. Linear VC Dimension.
  7. Examples
  8. Example 1 (Mixtures of product distributions).
  9. Example 2 (Bounded mutual information).
  10. Example 3 (Convex sets in $\mathbb R^d$).
  11. Example 4 (Permutation graphs: why new machinery is needed).
  12. Quantitative Bounds
  13. Upper and lower bounds
  14. Special case: product distributions.
  15. Tightness of the lower bound.
  16. ...and 29 more sections

Key Result

theorem 1

Let $\mathcal{X} = \mathcal{W}_1 \times \cdots \times \mathcal{W}_d$ be a product measurable space, and let $\mathcal{F} \subseteq 2^{\mathcal{X}}$ be a family of events. Then, the following are equivalent:

Figures (1)

  • Figure 1: Uniform estimation regimes. The dashed curve separates uniformly and non-uniformly estimable regimes based on the richness of the event class $\mathcal{F}$ and distribution family $\mathcal{P}$.

Theorems & Definitions (65)

  • definition 1: Uniform Estimability
  • definition 2: Uniform box--continuity
  • definition 3: Linear VC dimension
  • theorem 1: Main Result
  • proposition 1: Mixtures of product distributions
  • proposition 2: Bounded total correlation implies uniform box--continuity
  • lemma 1: Grid Sauer-Shelah-Perles bound, informal
  • proof
  • proposition 3: Uniform estimability under bounded linear VC dimension
  • proof : Proof of Proposition \ref{['prop:upper']}
  • ...and 55 more