ValUES: A Framework for Systematic Validation of Uncertainty Estimation in Semantic Segmentation

TL;DR

This work tackles the gap between uncertainty estimation theory and practical deployment for semantic segmentation by introducing ValUES, a framework for controlled evaluation, component-level ablations, and testbeds across five uncertainty applications. It formalizes a four-component model of uncertainty (C0–C3) and defines how aleatoric and epistemic uncertainties contribute to predictive uncertainty, with explicit measurement strategies. Through an empirical separation study, it shows that separation of AU and EU can hold in synthetic settings but does not always generalize to real data, and highlights the crucial role of aggregation and downstream-task selection; ensembles emerge as generally robust, while test-time augmentation provides a cost-effective alternative for EU. The findings offer practical recommendations to practitioners for configuring and evaluating uncertainty methods in real-world segmentation scenarios, promoting a systematic knowledge base in the field.

Abstract

Uncertainty estimation is an essential and heavily-studied component for the reliable application of semantic segmentation methods. While various studies exist claiming methodological advances on the one hand, and successful application on the other hand, the field is currently hampered by a gap between theory and practice leaving fundamental questions unanswered: Can data-related and model-related uncertainty really be separated in practice? Which components of an uncertainty method are essential for real-world performance? Which uncertainty method works well for which application? In this work, we link this research gap to a lack of systematic and comprehensive evaluation of uncertainty methods. Specifically, we identify three key pitfalls in current literature and present an evaluation framework that bridges the research gap by providing 1) a controlled environment for studying data ambiguities as well as distribution shifts, 2) systematic ablations of relevant method components, and 3) test-beds for the five predominant uncertainty applications: OoD-detection, active learning, failure detection, calibration, and ambiguity modeling. Empirical results on simulated as well as real-world data demonstrate how the proposed framework is able to answer the predominant questions in the field revealing for instance that 1) separation of uncertainty types works on simulated data but does not necessarily translate to real-world data, 2) aggregation of scores is a crucial but currently neglected component of uncertainty methods, 3) While ensembles are performing most robustly across the different downstream tasks and settings, test-time augmentation often constitutes a light-weight alternative. Code is at: https://github.com/IML-DKFZ/values

Figures (14)

• Figure 1: Framework for systematic validation of uncertainty methods in segmentation. With our framework, we aim to overcome pitfalls in the current validation of uncertainty methods for semantic segmentation by satisfying the three requirements (R1-R3) for a systematic validation: We explicitly control for aleatoric and epistemic uncertainty in the data and references (R1). We define and validate four individual components C0-C3 of uncertainty methods (R2): First, one or multiple segmentation outputs are generated by the segmentation backbone (C0) and the prediction model (C1). Next, an uncertainty measure is applied (C2) producing an uncertainty heatmap, which can be aggregated using an aggregation strategy (C3). Finally, the real-world capabilities of methods need to be validated on various downstream tasks (R3).
• Figure 2: a) General findings of the separation study. Green/red denotes agreement/disagreement with theoretical claims, orange represents partial agreement. b) Underlying quantitative results on the toy data set. Results show C2 and are aggregated over C1 and C3. Results for LIDC and GTA5/CS are displayed in \ref{['fig:main_results_barplot']} (see gray-shaded "Q" indicators). Details are in \ref{['appendix:separation']}
• Figure 3: Aggregated results showing improvements over the mean performance (higher is better) to assess general trends for each component (C1-C3) across settings for each dataset. Due to the averaging across different components (not seeds), high standard deviations are expected. Uncertainty measures not suited for a specific downstream task are excluded in the average, indicated with crosses in the color of the specific uncertainty measure. Detailed results are shown in \ref{['appendix:downstreamtask_tables']}. Metrics: OoD (AUROC), AL (% improvement over random), FD (AURC), CALIB (ACE), AM (NCC).
• Figure 4: Aleatoric data scenario. \ref{['fig:aleatoricTrainingDataMockup']} shows the input images in the training set, which are ambiguous due to Gaussian blur to the outside. \ref{['fig:aleatoricRater1DataMockup']} - \ref{['fig:aleatoricRater3DataMockup']} show three different reference ratings that are generated for the input images. \ref{['fig:aleatoricTestDataMockup']} shows test images and \ref{['fig:aleatoricExpectationDataMockup']} the expected uncertainty maps. The uncertainty regions are explained in \ref{['fig:aleatoricLegend']}.
• Figure 5: Epistemic data scenario. \ref{['fig:epistemicTrainingDataMockup']} shows the input images in the training set. \ref{['fig:epistemicGTDataMockup']} shows the ground truth segmentation. \ref{['fig:epistemicTestDataMockup']} shows test images that differ in various aspects from the training data. \ref{['fig:epistemicExpectationShapeDataMockup']} and \ref{['fig:epistemicExpectationIntensityDataMockup']} show possible uncertainty maps, depending on what the network learned. The uncertainty regions are explained in \ref{['fig:epistemicLegend']}.