Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS) challenge results

TL;DR

CURVAS addresses the challenge of reliable medical image segmentation under multi-rater uncertainty by benchmarking both segmentation accuracy and calibration. It introduces a multi-annotator ground truth framework, combines $DSC$, $cECE$, and $CRPS$ to evaluate accuracy, calibration, and volume estimates, and employs bootstrap and statistical analyses to assess ranking stability. The study shows that the best-performing models are not only accurate but also well-calibrated, with pretraining and diverse data contributing to robustness under distribution shifts. The findings highlight the necessity of uncertainty-aware, calibration-conscious evaluation for trustworthy clinical deployment and offer concrete guidance for future multi-annotator segmentation benchmarks.

Abstract

Deep learning (DL) has become the dominant approach for medical image segmentation, yet ensuring the reliability and clinical applicability of these models requires addressing key challenges such as annotation variability, calibration, and uncertainty estimation. This is why we created the Calibration and Uncertainty for multiRater Volume Assessment in multiorgan Segmentation (CURVAS), which highlights the critical role of multiple annotators in establishing a more comprehensive ground truth, emphasizing that segmentation is inherently subjective and that leveraging inter-annotator variability is essential for robust model evaluation. Seven teams participated in the challenge, submitting a variety of DL models evaluated using metrics such as Dice Similarity Coefficient (DSC), Expected Calibration Error (ECE), and Continuous Ranked Probability Score (CRPS). By incorporating consensus and dissensus ground truth, we assess how DL models handle uncertainty and whether their confidence estimates align with true segmentation performance. Our findings reinforce the importance of well-calibrated models, as better calibration is strongly correlated with the quality of the results. Furthermore, we demonstrate that segmentation models trained on diverse datasets and enriched with pre-trained knowledge exhibit greater robustness, particularly in cases deviating from standard anatomical structures. Notably, the best-performing models achieved high DSC and well-calibrated uncertainty estimates. This work underscores the need for multi-annotator ground truth, thorough calibration assessments, and uncertainty-aware evaluations to develop trustworthy and clinically reliable DL-based medical image segmentation models.

Figures (7)

• Figure 1: Age Distribution Across Groups and Dataset Splits. The left chart shows the distribution of cases across Groups A, B, and C by age range, while the right chart presents the dataset split into training, validation, and test sets. Group B and C cases are more frequent in older individuals, aligning with a higher pathology burden. The dataset split remains balanced, ensuring representation across age groups and pathological complexity.
• Figure 2: Visual example of a CRPS calculation. Left figure: Gaussian Probability Distribution function (PDF) blue line with the predicted volume (red line). Right figure: Gaussian Cumulative Distribution function (CDF) (blue line) with the predicted volume (red line) and its corresponding Heavyside representation and the CRPS area.
• Figure 3: Pairwise comparison of the different metrics across all algorithms. The diagonal shows the distributions of each metric, while the off-diagonal plots depict the relationships between pairs of metrics. Each color represents a specific algorithm, consistent with the color scheme used in the boxplot Figure \ref{['fig:metrics']}, ensuring direct comparability between the two visualizations.
• Figure 4: Boxplots of the evaluation metrics (DSC, Confidence, ECE, and CRPS) for all algorithms. Each box represents the distribution of metric values for a specific algorithm, with each color corresponding to a specific algorithm. In this plot the outliers are not considered. Teams ranked from best to worst in the final ranking are visualized from left to right on the x-axis of each plot.
• Figure 5: Qualitative example of the segmentation produced by the different algorithms (ordered left to right by overall ranking) fare shown for the three target structures: pancreas (rows 1–2), kidneys (rows 3–4), and liver (rows 5–6). For each structure, the top row displays the consensus ground truth (rows 1, 3, 5) and the corresponding binarized segmentations, while the bottom row presents the dissensus ground truth (rows 2, 4, 6) alongside the probabilistic predictions (uncertainty maps).