Performance uncertainty in medical image analysis: a large-scale investigation of confidence intervals

TL;DR

This study tackles the problem of performance uncertainty in medical imaging AI by systematically evaluating confidence-interval (CI) methods across segmentation and classification tasks. Using a large-scale, non-parametric simulation framework based on real benchmark distributions, the authors compare parametric CIs, bootstrap variants (percentile, basic, BCa), and concentration-inequality bounds, assessing coverage and width for a wide range of metrics and aggregation schemes. Key findings show that sample size requirements for reliable CIs span from tens to thousands, depend on the chosen metric and aggregation (macro vs micro), and differ between segmentation and classification; notably, percentile bootstrap is generally robust, while basic bootstrap underperforms and BCa can fail in some cases. The work provides a comprehensive, data-driven basis for guidelines on reporting performance uncertainty in medical imaging AI, highlighting when certain CI methods are trustworthy and how study design choices influence CI reliability and precision.

Abstract

Performance uncertainty quantification is essential for reliable validation and eventual clinical translation of medical imaging artificial intelligence (AI). Confidence intervals (CIs) play a central role in this process by indicating how precise a reported performance estimate is. Yet, due to the limited amount of work examining CI behavior in medical imaging, the community remains largely unaware of how many diverse CI methods exist and how they behave in specific settings. The purpose of this study is to close this gap. To this end, we conducted a large-scale empirical analysis across a total of 24 segmentation and classification tasks, using 19 trained models per task group, a broad spectrum of commonly used performance metrics, multiple aggregation strategies, and several widely adopted CI methods. Reliability (coverage) and precision (width) of each CI method were estimated across all settings to characterize their dependence on study characteristics. Our analysis revealed five principal findings: 1) the sample size required for reliable CIs varies from a few dozens to several thousands of cases depending on study parameters; 2) CI behavior is strongly affected by the choice of performance metric; 3) aggregation strategy substantially influences the reliability of CIs, e.g. they require more observations for macro than for micro; 4) the machine learning problem (segmentation versus classification) modulates these effects; 5) different CI methods are not equally reliable and precise depending on the use case. These results form key components for the development of future guidelines on reporting performance uncertainty in medical imaging AI.

Figures (31)

• Figure 1: Not all confidence interval (CI) methods can be universally trusted. Here, CI 2 is too wide and thus underconfident about the target value. CI 3 is unreliable as it completely misses the target value. CI 1 is both reliable and precise.
• Figure 2: Link between study characteristics and behavior of confidence intervals (CIs). Left panel: main characteristics of a medical imaging study. Right panel: key findings of our investigation, revealing how study characteristics impact the reliability and precision of CIs.
• Figure 3: Experimental design. One starts with a real benchmark instance: a given model applied to a given task. A kernel density estimation (KDE) is performed on the empirical distribution (metric for segmentation and logits for classification). Then, multiple (in our case 10,000) samples are drawn from this KDE. A CI is computed for each sample, providing 10,000 CIs. One then gets an empirical coverage value and average interval width. The process is repeated for all benchmark instances, sample sizes and CI methods. This provides the coverage (resp. width) behavior for a given metric and summary statistic. The process is further repeated to perform the full investigation.
• Figure 4: Comparison of different bootstrap methods. Coverage of CIs of the mean DSC (Panel A) and of the accuracy (Panel B) for different bootstrap CI methods. Other settings (summary statistics and metrics) are available in Supplementary Section \ref{['sec:full_results']}.
• Figure 5: BCa fails for CIs of the median. BCa’s coverage (orange) decreases with increasing sample size while that of percentile (purple) is adequate. Graphs for other segmentation metrics are shown in Supplementary Figure \ref{['fig:bca_all']}.