Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Auto-Prompting SAM for Weakly Supervised Landslide Extraction

Jian Wang, Xiaokang Zhang, Xianping Ma, Weikang Yu, Pedram Ghamisi

TL;DR

APSAM tackles weakly supervised landslide segmentation by auto-prompting the Segment Anything Model (SAM) with prompts derived from CAM-based object localization. It bypasses CAM refinement and SAM fine-tuning by introducing an adaptive prompt generation (APG) that creates box and point prompts from CAM heatmaps, enabling SAM to produce fine-grained pseudo-labels. The pseudo-labels train a semantic segmentation model, yielding improved OA, F1, and IoU on high-resolution Hong Kong and Turkey datasets, with IoU gains ranging from 3.69% to 16.41% over state-of-the-art weakly supervised methods. The approach demonstrates that carefully engineered prompts, anchored in CAM patterns, can effectively harness SAM for irregular, boundary-fragmented landslide objects in remote sensing imagery, offering a practical, scalable weakly supervised solution.

Abstract

Weakly supervised landslide extraction aims to identify landslide regions from remote sensing data using models trained with weak labels, particularly image-level labels. However, it is often challenged by the imprecise boundaries of the extracted objects due to the lack of pixel-wise supervision and the properties of landslide objects. To tackle these issues, we propose a simple yet effective method by auto-prompting the Segment Anything Model (SAM), i.e., APSAM. Instead of depending on high-quality class activation maps (CAMs) for pseudo-labeling or fine-tuning SAM, our method directly yields fine-grained segmentation masks from SAM inference through prompt engineering. Specifically, it adaptively generates hybrid prompts from the CAMs obtained by an object localization network. To provide sufficient information for SAM prompting, an adaptive prompt generation (APG) algorithm is designed to fully leverage the visual patterns of CAMs, enabling the efficient generation of pseudo-masks for landslide extraction. These informative prompts are able to identify the extent of landslide areas (box prompts) and denote the centers of landslide objects (point prompts), guiding SAM in landslide segmentation. Experimental results on high-resolution aerial and satellite datasets demonstrate the effectiveness of our method, achieving improvements of at least 3.0\% in F1 score and 3.69\% in IoU compared to other state-of-the-art methods. The source codes and datasets will be available at https://github.com/zxk688.

Auto-Prompting SAM for Weakly Supervised Landslide Extraction

TL;DR

APSAM tackles weakly supervised landslide segmentation by auto-prompting the Segment Anything Model (SAM) with prompts derived from CAM-based object localization. It bypasses CAM refinement and SAM fine-tuning by introducing an adaptive prompt generation (APG) that creates box and point prompts from CAM heatmaps, enabling SAM to produce fine-grained pseudo-labels. The pseudo-labels train a semantic segmentation model, yielding improved OA, F1, and IoU on high-resolution Hong Kong and Turkey datasets, with IoU gains ranging from 3.69% to 16.41% over state-of-the-art weakly supervised methods. The approach demonstrates that carefully engineered prompts, anchored in CAM patterns, can effectively harness SAM for irregular, boundary-fragmented landslide objects in remote sensing imagery, offering a practical, scalable weakly supervised solution.

Abstract

Weakly supervised landslide extraction aims to identify landslide regions from remote sensing data using models trained with weak labels, particularly image-level labels. However, it is often challenged by the imprecise boundaries of the extracted objects due to the lack of pixel-wise supervision and the properties of landslide objects. To tackle these issues, we propose a simple yet effective method by auto-prompting the Segment Anything Model (SAM), i.e., APSAM. Instead of depending on high-quality class activation maps (CAMs) for pseudo-labeling or fine-tuning SAM, our method directly yields fine-grained segmentation masks from SAM inference through prompt engineering. Specifically, it adaptively generates hybrid prompts from the CAMs obtained by an object localization network. To provide sufficient information for SAM prompting, an adaptive prompt generation (APG) algorithm is designed to fully leverage the visual patterns of CAMs, enabling the efficient generation of pseudo-masks for landslide extraction. These informative prompts are able to identify the extent of landslide areas (box prompts) and denote the centers of landslide objects (point prompts), guiding SAM in landslide segmentation. Experimental results on high-resolution aerial and satellite datasets demonstrate the effectiveness of our method, achieving improvements of at least 3.0\% in F1 score and 3.69\% in IoU compared to other state-of-the-art methods. The source codes and datasets will be available at https://github.com/zxk688.
Paper Structure (12 sections, 9 equations, 5 figures, 2 tables, 1 algorithm)

This paper contains 12 sections, 9 equations, 5 figures, 2 tables, 1 algorithm.

Table of Contents

  1. Introduction
  2. Methodology
  3. Object Localization
  4. Adaptive Prompts Generation
  5. Pseudo Labeling with SAM
  6. Training and Inference
  7. Experimental Results
  8. Data Description
  9. Experimental setup
  10. Results
  11. Analysis
  12. Conclusion

Figures (5)

  • Figure 1: Overview of the proposed APSAM for landslide extraction.
  • Figure 2: Landslide extraction results on the Hong Kong dataset. (a) Original image. (b) Ground truth. The results were obtained by (c) CDA, (d) CPN, (e) CSE, (f) FlipCAM, (g) S2C, (h) LGAGNet, and (i) Our Method. The image size is 256 × 256 pixels.
  • Figure 3: Landslide extraction results on the Turkey dataset. (a) Original image. (b) Ground truth. The results were obtained by (c) CDA, (d) CPN, (e) CSE, (f) FlipCAM, (g) S2C, (h) LGAGNet, and (i) Our Method. The image size is 256 × 256 pixels.
  • Figure 4: Comparison of prompting results obtained by SAM and APSAM. (a) Original image. (b) Ground truth. (c) SAM's point prompts. (d) Overlay result of SAM segmentation by color mask. (e) APSAM's box and point prompts. (f) Result of APSAM segmentation.
  • Figure 5: Parameter analysis of the threshold $T$.