Data-driven Error Estimation: Upper Bounding Multiple Errors without Class Complexity as Input

TL;DR

This work tackles the problem of constructing high-probability upper bounds on the maximum estimation error across a class of estimation tasks without requiring explicit bounds on class complexity. It introduces a data-driven framework that uses a defining dataset to form per-task estimates and a separate holdout error dataset to bound each individual error, then yields a bound on the maximum error via the maximum of these per-task bounds, independent of the class size. The core theory shows that this bound holds with probability at least $1-\delta$ and naturally adapts to the correlation structure of errors, enabling simultaneous confidence intervals, excess-risk estimation, and potentially optimal model sets without traditional complexity terms. Localization further tightens bounds for particular instances, and the paper provides practical recipes for constructing multiple mean CIs and for contextual-bandit pipelines that rely on error estimates. Overall, the approach offers a simple, flexible, and computation-efficient alternative to union bounds, bootstrap, or PAC-Bayes in large or infinite hypothesis spaces.

Abstract

Constructing confidence intervals that are simultaneously valid across a class of estimates is central for tasks such as multiple mean estimation, bounding generalization error in machine learning, and adaptive experimental design. We frame this as an "error estimation problem," where the goal is to determine a high-probability upper bound on the maximum error for a class of estimates. We propose an entirely data-driven approach that derives such bounds for both finite and infinite class settings, naturally adapting to a potentially unknown correlation structure of random errors. Notably, our method does not require class complexity as an input, overcoming a major limitation of existing approaches such as union bounding and bounds based on Talagrand's inequality. In this paper, we present our simple yet general solution and demonstrate its flexibility through applications ranging from constructing multiple simultaneously valid confidence intervals to optimizing exploration in contextual bandit algorithms.

Figures (3)

• Figure 1: In this plot, we bound maximum error across estimates quantile estimates. We use $n =10 000$ i.i.d. samples from a standard normal distribution to estimate an increasing number of quantiles. We use two methods to bound the maximum error, and plot the excess (bound - true maximum error) of error bounds vs. increasing number of quantiles estimated for each: traditional union bound (orange) and our error estimation method (blue).
• Figure 2: Comparison of excess risk bounds over a class of Neural Networks. We plot the empirical risk of $g_\text{def}$ on $S_\text{err}$ (blue), our (probabilistic) error estimation bound (orange) and the true excess risk (green) against increasing noise levels.
• Figure 3: Visualization of data allocation in epoch $\tau$, with $\lambda = 1/2$

Theorems & Definitions (25)

• Theorem 2.2: Data-Driven Upper Bound on Max Error
• proof
• Corollary 2.5
• Remark 3.1: $\hat{u}(S_\text{err}, h, \delta)$ is often a surrogate for $e_h$
• Remark 3.2: There is Room for Improvement
• Corollary 3.3
• proof
• Remark 3.4: Tightness compared to union bound
• Corollary 3.5
• Remark 3.6