Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Constructing Confidence Intervals for 'the' Generalization Error -- a Comprehensive Benchmark Study

Hannah Schulz-Kümpel, Sebastian Fischer, Roman Hornung, Anne-Laure Boulesteix, Thomas Nagler, Bernd Bischl

TL;DR

This work conducts the largest benchmark to date of model-agnostic confidence intervals for the generalization error, evaluating 13 CI methods across 19 DGPs with seven inducers and multiple losses. It provides a unified framework, neutral comparisons, and publicly available benchmarking resources to guide future method development. The study identifies Corrected Resampled-T, Conservative-Z, and Nested CV as robust performers, offering practical recommendations depending on data size and computational budgets. By analyzing coverage, interval width, and runtime, the paper informs principled decision-making for uncertainty quantification in predictive performance. The findings promote empirical evaluation alongside theoretical guarantees and support broader adoption of GE CI methods in practice.

Abstract

When assessing the quality of prediction models in machine learning, confidence intervals (CIs) for the generalization error, which measures predictive performance, are a crucial tool. Luckily, there exist many methods for computing such CIs and new promising approaches are continuously being proposed. Typically, these methods combine various resampling procedures, most popular among them cross-validation and bootstrapping, with different variance estimation techniques. Unfortunately, however, there is currently no consensus on when any of these combinations may be most reliably employed and how they generally compare. In this work, we conduct a large-scale study comparing CIs for the generalization error, the first one of such size, where we empirically evaluate 13 different CI methods on a total of 19 tabular regression and classification problems, using seven different inducers and a total of eight loss functions. We give an overview of the methodological foundations and inherent challenges of constructing CIs for the generalization error and provide a concise review of all 13 methods in a unified framework. Finally, the CI methods are evaluated in terms of their relative coverage frequency, width, and runtime. Based on these findings, we can identify a subset of methods that we would recommend. We also publish the datasets as a benchmarking suite on OpenML and our code on GitHub to serve as a basis for further studies.

Constructing Confidence Intervals for 'the' Generalization Error -- a Comprehensive Benchmark Study

TL;DR

This work conducts the largest benchmark to date of model-agnostic confidence intervals for the generalization error, evaluating 13 CI methods across 19 DGPs with seven inducers and multiple losses. It provides a unified framework, neutral comparisons, and publicly available benchmarking resources to guide future method development. The study identifies Corrected Resampled-T, Conservative-Z, and Nested CV as robust performers, offering practical recommendations depending on data size and computational budgets. By analyzing coverage, interval width, and runtime, the paper informs principled decision-making for uncertainty quantification in predictive performance. The findings promote empirical evaluation alongside theoretical guarantees and support broader adoption of GE CI methods in practice.

Abstract

When assessing the quality of prediction models in machine learning, confidence intervals (CIs) for the generalization error, which measures predictive performance, are a crucial tool. Luckily, there exist many methods for computing such CIs and new promising approaches are continuously being proposed. Typically, these methods combine various resampling procedures, most popular among them cross-validation and bootstrapping, with different variance estimation techniques. Unfortunately, however, there is currently no consensus on when any of these combinations may be most reliably employed and how they generally compare. In this work, we conduct a large-scale study comparing CIs for the generalization error, the first one of such size, where we empirically evaluate 13 different CI methods on a total of 19 tabular regression and classification problems, using seven different inducers and a total of eight loss functions. We give an overview of the methodological foundations and inherent challenges of constructing CIs for the generalization error and provide a concise review of all 13 methods in a unified framework. Finally, the CI methods are evaluated in terms of their relative coverage frequency, width, and runtime. Based on these findings, we can identify a subset of methods that we would recommend. We also publish the datasets as a benchmarking suite on OpenML and our code on GitHub to serve as a basis for further studies.
Paper Structure (68 sections, 1 theorem, 62 equations, 23 figures, 10 tables)

This paper contains 68 sections, 1 theorem, 62 equations, 23 figures, 10 tables.

Table of Contents

  1. Introduction
  2. Our Contribution
  3. 1. A comprehensive, neutral comparison
  4. 2. A foundation for evaluating future methods
  5. 3. A hypothesis-generating empirical study.
  6. Setting and notation
  7. Essential conceptual considerations
  8. The two targets of inference
  9. The role of resampling in estimating the generalization error
  10. Sources of uncertainty
  11. Theoretical validity
  12. Summary of existing methods
  13. Holdout
  14. Replace-One CV
  15. Repeated Replace-One CV
  16. ...and 53 more sections

Key Result

Lemma 1

Consider a random variable $\bm{Z}\sim\mu_Z$ and let $q_\alpha(\mu_Z)$, $\alpha\in(0,1)$, denote the $\alpha$-quantile of the symmetric probability distribution $\mu_Z$ of $\bm{Z}$, If, for some sequence of values in $\mathbb{R}$$(s_n)_{n\in\mathds{N}}$ it holds that then the interval $[\widehat{\Theta}_0(\mathcal{D}_n)\pm q_{1-\frac{\alpha}{2}}(\mu_Z)\hat{s}(\mathcal{D}_n)]$ is an asymptotically

Figures (23)

  • Figure 1: Pseudocode for the main experiment. Here, $\mathcal{A}_{\mathcal{D}, \mathcal{I}, \mathcal{L}}(\mathcal{J})$ denotes any CI method from \ref{['subchap:CImethods']} being applied to the problem instance $(\mathcal{D}, \mathcal{I}, \mathcal{L})$ given a specific split $\mathcal{J}$, which results in a point estimate $\widehat{GE}$ and CI borders $CI^{(L)}$, $CI^{(U)}$. $\mathcal{E}_{\mathcal{D}, \mathcal{I}, \mathcal{L}}(\mathcal{D}_\text{val}, \mathcal{J})$ denotes the calculation of a risk value and (element of) a PQ, if applicable.
  • Figure 2: Comparison of all (versions of) methods for computing CIs for the GE compared in this work. The highlighted methods were considered for further analysis.
  • Figure 3: Average coverage for the (versions of) methods that were not dismissed based on \ref{['fig:FirstRound']}, as a function of data size $n_\mathcal{D}$ and aggregated over DGPs and inducers. The upper rows contains those methods that were applied to all data sizes, and the lowest row those that were only applied to data sizes $100$ and $500$.
  • Figure 4: Average Coverage and median CI width (based only on classification tasks) for different configurations of the best-performing methods.
  • Figure 5: Comparison of the three best performing CI for GE methods: Corrected Resampled-T, Conservative-Z, and Nested CV. Due to computational cost, the MLP was only fit on data of size $5000$.
  • ...and 18 more figures

Theorems & Definitions (11)

  • Remark 1: Interpretation of risk and expected risk
  • Remark 2: Complexities inherent in resampling-based inference about the (expected) risk
  • Remark 3: Formal target quantities for the generalization error
  • Definition 1: Asymptotically exact coverage intervals
  • Remark 4
  • Remark 5: Missing scaling constant in variance estimate of austern2020asymptotics
  • Remark 6: The performance of Holdout-based CIs
  • Remark 7: CV Wald and Decision Trees
  • Remark 8: Empirical coverage of risk and expected risk
  • Lemma 1
  • ...and 1 more