Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

A unified framework for learning with nonlinear model classes from arbitrary linear samples

Ben Adcock, Juan M. Cardenas, Nick Dexter

TL;DR

A unified framework is introduced that accommodates objects in arbitrary Hilbert spaces, general (possibly vector-valued) random linear measurements and general types of nonlinear models and establishes novel learning guarantees for this framework that explicitly relate the required amount of data to structural properties of the model class, yielding near-optimal generalization bounds.

Abstract

We study the fundamental problem of learning an unknown object from data using a prescribed model class. We introduce a unified framework that accommodates objects in arbitrary Hilbert spaces, general (possibly vector-valued) random linear measurements and general types of nonlinear models. We establish novel learning guarantees for this framework that explicitly relate the required amount of data to structural properties of the model class, yielding near-optimal generalization bounds. A central concept we introduce is the variation of a model class relative to a distribution of sampling operators, which quantifies how the model interacts with the measurement process. Combined with entropy integrals that capture the model's complexity, this forms the foundation of our guarantees. Our framework is sufficiently general to recover and unify various well-known problems, such as matrix sketching, compressed sensing with isotropic measurements and compressed sensing with generative models. In each case, existing results arise as direct corollaries of our theory. For compressed sensing with generative models, we also derive the first guarantees for arbitrary Lipschitz generative maps combined with general linear measurements. Overall, our work provides a unified perspective on learning from general data and introduces novel theoretical guarantees that consolidate, sharpen and extend existing results.

A unified framework for learning with nonlinear model classes from arbitrary linear samples

TL;DR

A unified framework is introduced that accommodates objects in arbitrary Hilbert spaces, general (possibly vector-valued) random linear measurements and general types of nonlinear models and establishes novel learning guarantees for this framework that explicitly relate the required amount of data to structural properties of the model class, yielding near-optimal generalization bounds.

Abstract

We study the fundamental problem of learning an unknown object from data using a prescribed model class. We introduce a unified framework that accommodates objects in arbitrary Hilbert spaces, general (possibly vector-valued) random linear measurements and general types of nonlinear models. We establish novel learning guarantees for this framework that explicitly relate the required amount of data to structural properties of the model class, yielding near-optimal generalization bounds. A central concept we introduce is the variation of a model class relative to a distribution of sampling operators, which quantifies how the model interacts with the measurement process. Combined with entropy integrals that capture the model's complexity, this forms the foundation of our guarantees. Our framework is sufficiently general to recover and unify various well-known problems, such as matrix sketching, compressed sensing with isotropic measurements and compressed sensing with generative models. In each case, existing results arise as direct corollaries of our theory. For compressed sensing with generative models, we also derive the first guarantees for arbitrary Lipschitz generative maps combined with general linear measurements. Overall, our work provides a unified perspective on learning from general data and introduces novel theoretical guarantees that consolidate, sharpen and extend existing results.
Paper Structure (32 sections, 22 theorems, 199 equations)

This paper contains 32 sections, 22 theorems, 199 equations.

Table of Contents

  1. Introduction
  2. The framework
  3. Contributions
  4. Related work
  5. Outline
  6. Examples
  7. Sampling problems
  8. Model classes
  9. Key concepts
  10. Additional notation and approximate minimizers
  11. Shifted and difference sets, cones and projections on the unit sphere
  12. Covering numbers
  13. Variation
  14. Main results
  15. General result
  16. ...and 17 more sections

Key Result

Theorem 1.1

Consider the setup of § s:setup with $\alpha = \beta = 1$ and $\mathcal{A}_i = \mathcal{A}$, $\forall i$. Let $\mathbb{U}$ be a subset of a finite-dimensional subspace of $\mathbb{X}_0$ and suppose that $\Delta \mathbb{U} : = \mathbb{U} - \mathbb{U}$ is a cone. Suppose that, for some $0 < \epsilon < where $\tau = \sqrt{\frac{\Phi(S(\Delta \mathbb{U}) ; \bar{\mathcal{A}} )}{\Phi(S(\Delta^2 \mathbb{

Theorems & Definitions (55)

  • Theorem 1.1: Simplified main result
  • remark 1
  • Definition 3.1: $\gamma$- and $(\gamma,\theta)$-minimizers
  • Definition 3.2: Covering number
  • Definition 3.3: Variation with respect to a distribution
  • remark 2: Upper bounds for the variation
  • Definition 3.4: Constant distribution
  • Definition 3.5: Variation with respect to a collection
  • Theorem 4.1: General result
  • remark 3: Worst-case bound
  • ...and 45 more