Improving EO Foundation Models with Confidence Assessment for enhanced Semantic segmentation

TL;DR

The model, CAS, identifies segments with incorrect predicted labels using the proposed combined confidence metric, refines the model, and enhances its performance, showing that this strategy is effective and that the proposed model CAS outperforms other baseline models.

Abstract

Confidence assessments of semantic segmentation algorithms are important. Ideally, deep learning models should have the ability to predict in advance whether their output is likely to be incorrect. Assessing the confidence levels of model predictions in Earth Observation (EO) classification is essential, as it can enhance semantic segmentation performance and help prevent further exploitation of the results in case of erroneous prediction. The model we developed, Confidence Assessment for enhanced Semantic segmentation (CAS), evaluates confidence at both the segment and pixel levels, providing both labels and confidence scores as output. Our model, CAS, identifies segments with incorrect predicted labels using the proposed combined confidence metric, refines the model, and enhances its performance. This work has significant applications, particularly in evaluating EO Foundation Models on semantic segmentation downstream tasks, such as land cover classification using Sentinel-2 satellite data. The evaluation results show that this strategy is effective and that the proposed model CAS outperforms other baseline models.

Figures (6)

• Figure 1: CAS for confidence-aware segmentation: assign confidence metric to predictions, identify wrong predicted labels and refine the model.
• Figure 2: Semantic segmentation land cover classification, confidence metric estimation, and confidence assessment by the proposed model CAS on satellite Sentinel-2 data, using the dataset WorldCover. The main classes here are: Tree cover, Grassland, Cropland, and Built-up.
• Figure 3: Semantic segmentation and confidence estimation and assessment by the model CAS described in Sec. \ref{['sec:sectionProposedMethod']}, on Sentinel-2 image data using the dataset ESA WorldCover with $11$ land cover classes.
• Figure 4: Land cover semantic segmentation classification, and both confidence assignment and assessment, by the proposed model CAS on Sentinel-2 L2A data trained on the labelled dataset WorldCover.
• Figure 5: Histogram plot for CAS with Eq. \ref{['eq:equationnumbbeerreerr']} for semantic segmentation land cover classification on Sentinel-2 L2A multi-spectral data using WorldCover. The aim is to effectively separate misclassifications and correct classifications. The horizontal axis is the confidence metric.