Uncertainty evaluation of segmentation models for Earth observation

TL;DR

This work tackles the practical problem of estimating per-pixel uncertainty in semantic segmentation for Earth observation, benchmarking uncertainty methods across architectures and two diverse remote-sensing datasets. It proposes a pragmatic evaluation framework with three downstream tasks: identifying misclassified pixels (UE-PR), post-hoc performance improvement (PR on withheld uncertain pixels), and identifying noise-corrupted regions (Uncertainty-Noise curves), using both pixel- and image-level insights. The study compares probability-based metrics (normalized entropy and $1-\max_k \hat{p}_{ik}$), ensembles, and Stochastic Segmentation Networks (SSN) across three architectures (U-TAE, UNET3D, TSViT), revealing that Transformer-based TSViT generally yields superior uncertainty identification, while ensembles provide the best segmentation accuracy; SSN offers a compelling balance with robustness to input noise. The authors advocate standardized uncertainty evaluation via a dedicated test set and highlight practical guidance for remote-sensing deployments, including directions for future research such as superpixel-level uncertainty and exploring diverse loss functions to improve calibration.

Abstract

This paper investigates methods for estimating uncertainty in semantic segmentation predictions derived from satellite imagery. Estimating uncertainty for segmentation presents unique challenges compared to standard image classification, requiring scalable methods producing per-pixel estimates. While most research on this topic has focused on scene understanding or medical imaging, this work benchmarks existing methods specifically for remote sensing and Earth observation applications. Our evaluation focuses on the practical utility of uncertainty measures, testing their ability to identify prediction errors and noise-corrupted input image regions. Experiments are conducted on two remote sensing datasets, PASTIS and ForTy, selected for their differences in scale, geographic coverage, and label confidence. We perform an extensive evaluation featuring several models, such as Stochastic Segmentation Networks and ensembles, in combination with a number of neural architectures and uncertainty metrics. We make a number of practical recommendations based on our findings.

Figures (22)

• Figure 1: Results from SSN on ForTy. Top row from left to right: input image, segmentation labels, predicted segmentation, model uncertainty (Sampled Categorical Variation), segmentation mistakes. Middle and Bottom rows: alternative predictions sampled from the model. Note that the dark blue segments represent the background class and are ignored.
• Figure 2: Distribution of the proportion of corrupted pixels per image, computed on the PASTIS test set. Left: For object-level corruption; Right: For elliptical noise.
• Figure 3: Uncertainty-Error Precision-Recall curves for identifying segmentation mistakes on PASTIS for different architecture / model type combinations. Left: Deterministic models. Middle: SSNs. Right: Ensembles. Transformers clearly outperform other architectures.
• Figure 4: TSViT on PASTIS. Left: Uncertainty-Error Precision-Recall curves for the task of identifying segmentation mistakes. Right: Precision-Recall curves for the original segmentation task when using uncertainty estimates to reject pixels.
• Figure 5: TSViT on ForTy. Left: Uncertainty-Error Precision-Recall curves for the downstream task of identifying segmentation mistakes. Right: Precision-Recall curves for the original segmentation task when using uncertainty estimates to reject pixels.