A Review of Bayesian Uncertainty Quantification in Deep Probabilistic Image Segmentation

TL;DR

The paper addresses the fragmentation in uncertainty quantification for deep probabilistic image segmentation by proposing a unified Bayesian framework that distinguishes feature- and parameter-level uncertainty and links them to four core tasks: observer variability, active learning, model introspection, and model generalization. It surveys perceptual models (pixel-level independence vs spatial coherence, latent-variable VAEs/GANs/DDPMs) and parameter-space approaches (VI, Laplace, TTA), highlighting their trade-offs, calibration issues, and scalability to 2D/3D data. Key contributions include a structured taxonomy, critical discussion of spatial aggregation, and practical guidelines for method selection, evaluation, and reproducibility, complemented by a call for standardized benchmarks and broader coverage of multiclass, instance, and panoptic segmentation. The work emphasizes the need for reliable, explainable, actionable, and unbiased uncertainty in real-world deployments, and identifies uncertainty-aware active learning, data-driven benchmarks, and transformer-based backbones as promising directions. Overall, the paper provides a cohesive roadmap to advance robust, interpretable, and efficient segmentation systems that can operate safely in high-stakes environments.

Abstract

Advances in architectural design, data availability, and compute have driven remarkable progress in semantic segmentation. Yet, these models often rely on relaxed Bayesian assumptions, omitting critical uncertainty information needed for robust decision-making. Despite growing interest in probabilistic segmentation to address point-estimate limitations, the research landscape remains fragmented. In response, this review synthesizes foundational concepts in uncertainty modeling, analyzing how feature- and parameter-distribution modeling impact four key segmentation tasks: Observer Variability, Active Learning, Model Introspection, and Model Generalization. Our work establishes a common framework by standardizing theory, notation, and terminology, thereby bridging the gap between method developers, task specialists, and applied researchers. We then discuss critical challenges, including the nuanced distinction between uncertainty types, strong assumptions in spatial aggregation, the lack of standardized benchmarks, and pitfalls in current quantification methods. We identify promising avenues for future research, such as uncertainty-aware active learning, data-driven benchmarks, transformer-based models, and novel techniques to move from simple segmentation problems to uncertainty in holistic scene understanding. Based on our analysis, we offer practical guidelines for researchers on method selection, evaluation, reproducibility, and meaningful uncertainty estimation. Ultimately, our goal is to facilitate the development of more reliable, efficient, and interpretable segmentation models that can be confidently deployed in real-world applications.

Figures (15)

• Figure 1: Overview of the sections. The leafs of the presented hierarchical tree are related to subsections of the article.
• Figure 2: Uncertainty modeling by treating pixel-level confidence values as parameters of a Probability Mass Function.
• Figure 3: Illustration of the likelihood function in segmentation models. Color intensities reflect the normalized confidence values that can be interpreted as probabilities. (a) The continuous output results in a horizontal gradient. (b) Maximum likelihood thresholding can be applied. (c) However, the coherence of the segmentation suffers when sampling because information on neighboring pixels is disregarded.
• Figure 4: Visualization of model calibration with a reliability diagram, which demonstrates whether the predicted model confidences align with the observed empirical accuracies.
• Figure 5: Illustration of the PixelCNN-based PixelSeg zhang_pixelseg_2022. Parameter 'C' indicates the concatenation module, 'R' the resampling module and $\sigma$ the softmax activation. Dotted elements appear during test-time sampling.