Clinical Uncertainty Impacts Machine Learning Evaluations

TL;DR

The paper addresses the problem that clinical annotation uncertainty is often ignored in model evaluation, leading to potentially misleading rankings. It proposes uncertainty-aware soft metrics that operate directly on probabilistic labels, with closed-form, linear-time formulas for $s\text{-}AUROC$ and $s\text{-}AP$ built from $p_i \in [0,1]$ and cumulative counts $n_i^+$, $n_i^-$, yielding robust, distribution-aware evaluations. Through experiments on multiple medical-imaging benchmarks, the authors demonstrate that soft metrics can reweight ambiguous cases, reshape model rankings, and improve stability via bootstrap analysis, while remaining agnostic to the annotation generation process. They advocate for releasing unaggregated annotations and adopting uncertainty-aware evaluation in practice, highlighting broader applicability beyond imaging and discussing data-sharing, privacy, and fairness considerations. Overall, the work provides practical, scalable tools and a strong case for integrating annotation uncertainty into standard evaluation pipelines to better reflect clinical realities.

Abstract

Clinical dataset labels are rarely certain as annotators disagree and confidence is not uniform across cases. Typical aggregation procedures, such as majority voting, obscure this variability. In simple experiments on medical imaging benchmarks, accounting for the confidence in binary labels significantly impacts model rankings. We therefore argue that machine-learning evaluations should explicitly account for annotation uncertainty using probabilistic metrics that directly operate on distributions. These metrics can be applied independently of the annotations' generating process, whether modeled by simple counting, subjective confidence ratings, or probabilistic response models. They are also computationally lightweight, as closed-form expressions have linear-time implementations once examples are sorted by model score. We thus urge the community to release raw annotations for datasets and to adopt uncertainty-aware evaluation so that performance estimates may better reflect clinical data.

Figures (6)

• Figure 1: Comparison of ordinary and soft metrics on three datasets with three tasks each. In the left panels, dots represent different backbones, and dashed lines indicate Pearson score correlations whose $R^2$ values are reported on the right. Details are in Appendix Table \ref{['tab:OrdinaryVSSoftMetrics']}.
• Figure 2: Results for data quality issue detection on CleanPatrick. Bars show the uncertainty-aware score s-AP, and vertical ticks mark the corresponding AP. Detailed results are in Table \ref{['tab:OrdinaryVSSoftMetrics']} of the appendix.
• Figure 3: Average correlation coefficient of rankings produced by ordinary and soft metrics upon annotation bootstrap across 13 tasks. Details are in Figure \ref{['fig:bootstrap']} of the appendix.
• Figure 4: Distribution of soft labels for the evaluated datasets. Dotted lines represent the threshold used to produce the hard labels, and the colors represent binary positive and negative labels.
• Figure 5: Correlation coefficient of model rankings produced by both ordinary and soft metrics averaged over 1,000 bootstrapped samples, which represent random annotation draws.

Theorems & Definitions (2)

• definition 1: Soft auroc
• definition 2: Soft ap