A multi-scale vision transformer-based multimodal GeoAI model for mapping Arctic permafrost thaw

TL;DR

The paper tackles the challenge of mapping small, boundary-ambiguous Retrogressive Thaw Slumps (RTS) across the Arctic by introducing a multimodal instance segmentation framework built on Cascade Mask R-CNN with a multi-scale Vision Transformer backbone. It contributes two innovations: a residual cross-modality attention fusion module for efficient cross-modal feature integration and a two-stage training workflow with unimodal pretraining followed by multimodal fine-tuning to reduce memory demands while preserving accuracy. Empirical results show the proposed fusion strategy outperforms data-level and other feature-level fusions, and the unimodal pretraining approach yields memory savings with comparable or improved performance; the method also generalizes to other landform datasets such as GeoImageNet. The proposed approach enables accurate, scalable RTS delineation on pan-Arctic scales, aiding permafrost thaw monitoring, ecological and hydrological assessments, and climate-related impact studies, with future work focused on incorporating ArcticDEM and operational deployment.

Abstract

Retrogressive Thaw Slumps (RTS) in Arctic regions are distinct permafrost landforms with significant environmental impacts. Mapping these RTS is crucial because their appearance serves as a clear indication of permafrost thaw. However, their small scale compared to other landform features, vague boundaries, and spatiotemporal variation pose significant challenges for accurate detection. In this paper, we employed a state-of-the-art deep learning model, the Cascade Mask R-CNN with a multi-scale vision transformer-based backbone, to delineate RTS features across the Arctic. Two new strategies were introduced to optimize multimodal learning and enhance the model's predictive performance: (1) a feature-level, residual cross-modality attention fusion strategy, which effectively integrates feature maps from multiple modalities to capture complementary information and improve the model's ability to understand complex patterns and relationships within the data; (2) pre-trained unimodal learning followed by multimodal fine-tuning to alleviate high computing demand while achieving strong model performance. Experimental results demonstrated that our approach outperformed existing models adopting data-level fusion, feature-level convolutional fusion, and various attention fusion strategies, providing valuable insights into the efficient utilization of multimodal data for RTS mapping. This research contributes to our understanding of permafrost landforms and their environmental implications.

Figures (8)

• Figure 1: Study Areas. There are 855 RTS samples in total located in seven sites from both Canada and Russia. In Canada, study sites include Herschel Island (41 RTS samples), Horton Delta (43 RTS samples), Tuktoyaktuk peninsulas (165 RTS samples), and Banks Island (174 RTS samples). In Russia, 399 RTSs are from Yamal and Gydan peninsulas, 7 samples are near the Lena River, and the remaining 26 RTS samples are from Kolguev Island.
• Figure 2: Architecture of multimodal instance segmentation model for RTS mapping.
• Figure 3: Example multimodal training data and labels (in red) for RTS segmentation.
• Figure 4: Statistics on model parameters, memory consumption and accuracy (measured by average precision).
• Figure 5: Design and implementation of different fusion strategies. (a) Data-level fusion through channel expansion. (b) Feature-level convolutional fusion. Conv2D means a 2D convolutional layer. (c) Stacked-modality attention fusion. Q, K, and V represent the query, key, and value embeddings for attention calculation.