Quantifying Uncertainty and Variability in Machine Learning: Confidence Intervals for Quantiles in Performance Metric Distributions
Christoph Lehmann, Yahor Paromau
TL;DR
The paper reframes ML model evaluation as a study of performance metric distributions rather than single-point estimates, driven by sources of variation such as seed initialization and hyperparameter search. It advocates quantile-based analysis and confidence intervals for quantiles to quantify uncertainty, offering three nonparametric CI methods (exact, asymptotic, semiparametric bootstrap) plus a $t$-interval benchmark for the mean. Through simulations and real-data use cases (classification and regression), it shows that nonparametric exact CIs deliver strong coverage, middle quantiles are estimated more reliably, and semiparametric bootstrap can assist in very small samples though tail behavior may be challenging; practical guidance suggests $n\in[15,25]$ suffices for quantiles up to 90%. Overall, the framework enables robust, distribution-aware comparisons of training configurations and can inform data quality improvements and decision-making under uncertainty in ML pipelines.
Abstract
Machine learning models are widely used in applications where reliability and robustness are critical. Model evaluation often relies on single-point estimates of performance metrics such as accuracy, F1 score, or mean squared error, that fail to capture the inherent variability in model performance. This variability arises from multiple sources, including train-test split, weights initialization, and hyperparameter tuning. Investigating the characteristics of performance metric distributions, rather than focusing on a single point only, is essential for informed decision-making during model selection and optimization, especially in high-stakes settings. How does the performance metric vary due to intrinsic uncertainty in the selected modeling approach? For example, train-test split is modified, initial weights for optimization are modified or hyperparameter tuning is done using an algorithm with probabilistic nature? This is shifting the focus from identifying a single best model to understanding a distribution of the performance metric that captures variability across different training conditions. By running multiple experiments with varied settings, empirical distributions of performance metrics can be generated. Analyzing these distributions can lead to more robust models that generalize well across diverse scenarios. This contribution explores the use of quantiles and confidence intervals to analyze such distributions, providing a more complete understanding of model performance and its uncertainty. Aimed at a statistically interested audience within the machine learning community, the suggested approaches are easy to implement and apply to various performance metrics for classification and regression problems. Given the often long training times in ML, particular attention is given to small sample sizes (in the order of 10-25).