Logo
ScienceStack
Table of Contents
Fetching ...
Sign In

Segment Anything Model Can Not Segment Anything: Assessing AI Foundation Model's Generalizability in Permafrost Mapping

Wenwen Li, Chia-Yu Hsu, Sizhe Wang, Yezhou Yang, Hyunho Lee, Anna Liljedahl, Chandi Witharana, Yili Yang, Brendan M. Rogers, Samantha T. Arundel, Matthew B. Jones, Kenton McHenry, Patricia Solis

TL;DR

The paper evaluates the Segment Anything Model (SAM) as a GeoAI foundation model for permafrost mapping, examining zero-shot, knowledge-embedded, and fine-tuned pipelines on ice-wedge polygons and retrogressive thaw slumps, with EuroCrop for generalization. It finds that SAM's zero-shot segmentation on natural landscape features is limited, but performance improves substantially when given precise location priors or when fine-tuned on domain data; however, it generally lags behind state-of-the-art supervised methods, especially for complex RTS features. The authors propose a practical, detector-assisted workflow and a reusable framework to assess foundation models in geospatial tasks, plus recommendations for expanding natural-feature datasets and integrating spectral data. The work highlights both the potential and current limitations of SAM for AI-augmented terrain mapping and outlines directions for future research.

Abstract

This paper assesses trending AI foundation models, especially emerging computer vision foundation models and their performance in natural landscape feature segmentation. While the term foundation model has quickly garnered interest from the geospatial domain, its definition remains vague. Hence, this paper will first introduce AI foundation models and their defining characteristics. Built upon the tremendous success achieved by Large Language Models (LLMs) as the foundation models for language tasks, this paper discusses the challenges of building foundation models for geospatial artificial intelligence (GeoAI) vision tasks. To evaluate the performance of large AI vision models, especially Meta's Segment Anything Model (SAM), we implemented different instance segmentation pipelines that minimize the changes to SAM to leverage its power as a foundation model. A series of prompt strategies was developed to test SAM's performance regarding its theoretical upper bound of predictive accuracy, zero-shot performance, and domain adaptability through fine-tuning. The analysis used two permafrost feature datasets, ice-wedge polygons and retrogressive thaw slumps because (1) these landform features are more challenging to segment than manmade features due to their complicated formation mechanisms, diverse forms, and vague boundaries; (2) their presence and changes are important indicators for Arctic warming and climate change. The results show that although promising, SAM still has room for improvement to support AI-augmented terrain mapping. The spatial and domain generalizability of this finding is further validated using a more general dataset EuroCrop for agricultural field mapping. Finally, we discuss future research directions that strengthen SAM's applicability in challenging geospatial domains.

Segment Anything Model Can Not Segment Anything: Assessing AI Foundation Model's Generalizability in Permafrost Mapping

TL;DR

The paper evaluates the Segment Anything Model (SAM) as a GeoAI foundation model for permafrost mapping, examining zero-shot, knowledge-embedded, and fine-tuned pipelines on ice-wedge polygons and retrogressive thaw slumps, with EuroCrop for generalization. It finds that SAM's zero-shot segmentation on natural landscape features is limited, but performance improves substantially when given precise location priors or when fine-tuned on domain data; however, it generally lags behind state-of-the-art supervised methods, especially for complex RTS features. The authors propose a practical, detector-assisted workflow and a reusable framework to assess foundation models in geospatial tasks, plus recommendations for expanding natural-feature datasets and integrating spectral data. The work highlights both the potential and current limitations of SAM for AI-augmented terrain mapping and outlines directions for future research.

Abstract

This paper assesses trending AI foundation models, especially emerging computer vision foundation models and their performance in natural landscape feature segmentation. While the term foundation model has quickly garnered interest from the geospatial domain, its definition remains vague. Hence, this paper will first introduce AI foundation models and their defining characteristics. Built upon the tremendous success achieved by Large Language Models (LLMs) as the foundation models for language tasks, this paper discusses the challenges of building foundation models for geospatial artificial intelligence (GeoAI) vision tasks. To evaluate the performance of large AI vision models, especially Meta's Segment Anything Model (SAM), we implemented different instance segmentation pipelines that minimize the changes to SAM to leverage its power as a foundation model. A series of prompt strategies was developed to test SAM's performance regarding its theoretical upper bound of predictive accuracy, zero-shot performance, and domain adaptability through fine-tuning. The analysis used two permafrost feature datasets, ice-wedge polygons and retrogressive thaw slumps because (1) these landform features are more challenging to segment than manmade features due to their complicated formation mechanisms, diverse forms, and vague boundaries; (2) their presence and changes are important indicators for Arctic warming and climate change. The results show that although promising, SAM still has room for improvement to support AI-augmented terrain mapping. The spatial and domain generalizability of this finding is further validated using a more general dataset EuroCrop for agricultural field mapping. Finally, we discuss future research directions that strengthen SAM's applicability in challenging geospatial domains.
Paper Structure (11 sections, 5 figures, 6 tables)

This paper contains 11 sections, 5 figures, 6 tables.

Table of Contents

  1. Introduction
  2. AI foundation models for GeoAI vision tasks: Recent progress
  3. Data and experiments
  4. Datasets
  5. Experimental setup
  6. Zero-shot instance segmentation (Strategy 1)
  7. Knowledge-embedded instance segmentation with SAM (Strategies 2 - 4)
  8. Discussions
  9. Summary analysis
  10. Spatial and domain generalizability test
  11. Conclusions and future research directions

Figures (5)

  • Figure 1: Architecture of SAM (left of the dashed line) and CLIP (right of the dashed line) and their combined workflow for instance segmentation.
  • Figure 2: Results of zero-shot learning with the integrated SAM+CLIP model. The last column displays the final result, and the second-to-last column presents the intermediate results from SAM, which are used as input for CLIP. Image source: Maxar.com.
  • Figure 3: Results from knowledge-embedded learning with SAM. The results are those after fine-tuning. The images are the same as those in Figure \ref{['fig_sam_clip_result']}. Image source: Maxar.com.
  • Figure 4: Agricultural field mapping results using zero-shot learning with the integrated SAM+CLIP model. The last column displays the final result, and the second-to-last column presents the intermediate results from SAM, which are used as input for CLIP. Image source: Sentinel.
  • Figure 5: Agricultural field mapping results from knowledge-embedded learning with SAM. The results are those after fine-tuning. The images are the same as those in Figure \ref{['fig_sam_clip_result_agr']}.