SAM Guided Semantic and Motion Changed Region Mining for Remote Sensing Change Captioning

TL;DR

This work tackles remote sensing change captioning by integrating a Segment Anything Model–driven, region-aware analysis with a knowledge-graph prior to guide caption generation. The SAGE-CC framework combines a Bi-Temporal Scene Consistency Encoder, SAM-based motion and semantic region mining, and a Relational Knowledge Graph Reasoner, all fused in a Change-Aware Transformer decoder. Empirical results on LEVIR-CC, Dubai-CC, and WHU-CDC show state-of-the-art performance and robust ablations confirm the contribution of motion/semantic region mining and KG priors. The approach enhances both the factual accuracy and semantic coherence of captions, enabling more interpretable and actionable descriptions for remote sensing change detection tasks.

Abstract

Remote sensing change captioning is an emerging and popular research task that aims to describe, in natural language, the content of interest that has changed between two remote sensing images captured at different times. Existing methods typically employ CNNs/Transformers to extract visual representations from the given images or incorporate auxiliary tasks to enhance the final results, with weak region awareness and limited temporal alignment. To address these issues, this paper explores the use of the SAM (Segment Anything Model) foundation model to extract region-level representations and inject region-of-interest knowledge into the captioning framework. Specifically, we employ a CNN/Transformer model to extract global-level vision features, leverage the SAM foundation model to delineate semantic- and motion-level change regions, and utilize a specially constructed knowledge graph to provide information about objects of interest. These heterogeneous sources of information are then fused via cross-attention, and a Transformer decoder is used to generate the final natural language description of the observed changes. Extensive experimental results demonstrate that our method achieves state-of-the-art performance across multiple widely used benchmark datasets. The source code of this paper will be released on https://github.com/Event-AHU/SAM_ChangeCaptioning

Figures (6)

• Figure 1: Comparison between (a) existing change captioning framework and (b) our newly proposed SAM guided semantic and motion changed region mining for RS-CC.
• Figure 2: An illustration of our proposed remote sensing change captioning by SAM guided semantic, motion changed region mining and knowledge graph.
• Figure 3: Illustration of the built knowledge graph for the remote sensing change captioning task. Zoom in for better visualization.
• Figure 4: Remote sensing change image pairs and their corresponding ground-truth annotations are provided in the LEVIR-CC dataset. The matching parts are shown in green, and the mismatched parts are shown in red.
• Figure 5: Semantic segmentation results obtained using the SAM model. Blue, green, and pink regions denote building, vegetation, and road, respectively.