Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Estimating Uncertainty in Landslide Segmentation Models

Savinay Nagendra, Chaopeng Shen, Daniel Kifer

TL;DR

Several methods for assessing pixel-level uncertainty of the segmentation of landslide segmentation from satellite imagery are evaluated, including Pre-Threshold activations, Monte-Carlo Dropout and Test-Time Augmentation -- a method that measures the robustness of predictions in the face of data augmentation.

Abstract

Landslides are a recurring, widespread hazard. Preparation and mitigation efforts can be aided by a high-quality, large-scale dataset that covers global at-risk areas. Such a dataset currently does not exist and is impossible to construct manually. Recent automated efforts focus on deep learning models for landslide segmentation (pixel labeling) from satellite imagery. However, it is also important to characterize the uncertainty or confidence levels of such segmentations. Accurate and robust uncertainty estimates can enable low-cost (in terms of manual labor) oversight of auto-generated landslide databases to resolve errors, identify hard negative examples, and increase the size of labeled training data. In this paper, we evaluate several methods for assessing pixel-level uncertainty of the segmentation. Three methods that do not require architectural changes were compared, including Pre-Threshold activations, Monte-Carlo Dropout and Test-Time Augmentation -- a method that measures the robustness of predictions in the face of data augmentation. Experimentally, the quality of the latter method was consistently higher than the others across a variety of models and metrics in our dataset.

Estimating Uncertainty in Landslide Segmentation Models

TL;DR

Several methods for assessing pixel-level uncertainty of the segmentation of landslide segmentation from satellite imagery are evaluated, including Pre-Threshold activations, Monte-Carlo Dropout and Test-Time Augmentation -- a method that measures the robustness of predictions in the face of data augmentation.

Abstract

Landslides are a recurring, widespread hazard. Preparation and mitigation efforts can be aided by a high-quality, large-scale dataset that covers global at-risk areas. Such a dataset currently does not exist and is impossible to construct manually. Recent automated efforts focus on deep learning models for landslide segmentation (pixel labeling) from satellite imagery. However, it is also important to characterize the uncertainty or confidence levels of such segmentations. Accurate and robust uncertainty estimates can enable low-cost (in terms of manual labor) oversight of auto-generated landslide databases to resolve errors, identify hard negative examples, and increase the size of labeled training data. In this paper, we evaluate several methods for assessing pixel-level uncertainty of the segmentation. Three methods that do not require architectural changes were compared, including Pre-Threshold activations, Monte-Carlo Dropout and Test-Time Augmentation -- a method that measures the robustness of predictions in the face of data augmentation. Experimentally, the quality of the latter method was consistently higher than the others across a variety of models and metrics in our dataset.
Paper Structure (16 sections, 3 equations, 7 figures, 4 tables)

This paper contains 16 sections, 3 equations, 7 figures, 4 tables.

Table of Contents

  1. Introduction
  2. Related Work
  3. Landslide Dataset
  4. Comparative Study
  5. Quantitative Comparison
  6. Qualitative Comparison
  7. Representing Confidence
  8. Pre-Threshold Maps
  9. Monte-Carlo Dropout Maps
  10. Test-Time Augmentation Maps
  11. Evaluation Criteria
  12. Calibration Plots
  13. Area Under Curve (AUC)
  14. Image-Specific Thresholding
  15. Results
  16. ...and 1 more sections

Figures (7)

  • Figure 1: (Left to right) Example of bi-temporal image from our dataset, human label, prediction of our best model (Section \ref{['sec:model']}) and the corresponding TTA Confidence Map (Section \ref{['sec:uncertainty']})
  • Figure 2: Distribution of verified space-visible landslides in the dataset.
  • Figure 3: Qualitative results of U-Net (Multi) (best performing model) on example images from our dataset along with the three confidence maps evaluated.
  • Figure 4: Qualitative results of U-Net (single-image and multi-image task)
  • Figure 5: Qualitative comparison of the models
  • ...and 2 more figures